invention

After thinking about de-nuctifying the world I've started to investigate creating a Pidgin (formerly Gaim) plugin to control Skype. This means you're still using a closed network, but from an open program, which makes migrating easier (and allows people to try multiple networks from the same interface).

The Skype API docs are pretty good and it looks like the API can support at least controlling status, sending and receiving text messages, and making and receiving phone calls. So in principle a Skype plugin for Pidgin is a possibility.

There are a few docs on writing libpurple plugins for Pidgin but less comprehensively - the author of the Sametime plugin or the guy working on a MySpaceIM plugin probably know exactly how it works (the MySpaceIM plugin is a Google Summer of Code project that fits in with my idea very nicely). There is a C Plugin HOWTO in the source code which should get things started. Some ideas in the Adium Skype plugin bug as well.

On the way I read Silver Needle in the Skype which is an article on reverse-engineering Skype and using it - very interesting tech reading... but a shame they haven't made the code available

Well since it's 8 days till we leave Cape Town and head to Japan I'm not planning to actually do anything about this, but thought I'd write it up so I don't lose the links...

PS This blog doesn't currently have a comment mechanism so email comments are welcome to davidf at sjsoft dot com.

I've had a strange experience over the past year, and it's getting more common. As a developer, I used to be the one recommending software/tech things to my friends. Now all my non-technical friends have started recommending things to me that they have discovered on the intar-web. The trouble is, they're almost all closed systems - apparently free, but under the control of one group.

It started with Skype. Then xanga. And most recently it's been facebook. The speed with which facebook has spread amongst various of our friends is impressive. Some things about it are really good too - they're beginning to understand the sorts of things you can do with the Web.

But it's hard to explain that these sorts of things are the point of the Web as a Whole, rather than being a neat idea that can only work on a particular site. No-one understands the negative effects of a closed network. Or even more, the potential positive effects they're missing. Part of the problem is vocabulary - if you don't even have words to encapsulate the concepts to communicate, it's hard to argue for something. Open and closed networks are clear concepts to me but it's nice to embellish them somewhat.

So here's my attempt: A closed network utility under the control of one group that doesn't let you federate is called a nuct (a Network Utility Control Trap - don't let them innuct you into it). If you encounter one, you need a way of interacting with it that doesn't suck you in. The point of closed systems is of course, to prevent this.

But no-one seems to be doing the work required to break these things open We need a nuct-cracker suite (drum roll)

A simple start on easy ways to start breaking things open:

• open APIs to access data locked up in closed networks
  • a suite for different kinds of apps
  • social networks - same API to different ones
• plugins to open source open standards programs to interoperate with those networks
  • e.g. plugin to Gaim to control Skype
• open source servers with open APIs to replace the closed networks
• deploy and let the network effect take place

The trouble with tech pseudo-values: they're not the gospel. So on the one hand there comes a limit to avoiding things that would help good friendships because I think they could be done better. But more simply, I don't have enough time to actually create the online world the way I want it to be...

Is anyone doing something like this?

Eike Rathke points out the new Sun Weblog Publisher you can buy for $9.95 for blogging from within OpenOffice.org Writer (or StarOffice, not that I know anyone who has that :-)).

Sounds remarkably familiar. I wonder if Caolan has thought of trying to sell oooblogger...

Well seeing as Google has release a Code Search on open source code, I thought I would put it to good use: searching for pootle in filenames ending with .po helps you to quickly find some projects that are using Pootle to edit their PO files, and you can even see which version

Another nice way of showing how Pootle is gaining traction...

The Code Breakers looks like an interesting documentary on open source software around the world...

But the description contains an hilarious quote: According to Jonathan Murray of Microsoft "The Open Source community stimulates innovation in software, it's something that frankly we feel very good about and it's something that we absolutely see as being a partnership with Microsoft."

Very kind of them to include us as their partners voluntarily, I must say...

Gervase Markham has a nice rant on how complicated SIP is.

Jingle on Jabber is a much nicer solution if you're not a telecommunications person yourself. It's an open standard that's much simpler than SIP which makes it easier to implement, and there are open source libraries available that provide support, and it is Jabberish which is sensible and makes all the confederation work nicely. It addresses some of the technical issues that make people seem to like Skype (getting through firewalls etc) without having some of its headaches (proxying other people's phone calls through your computer, a totally mad idea).

The main issue is that the only current final-release program available with support is Google Talk; it's not open source and its only available on Windows.

I'm currently recommending Google Talk to Windows-using friends in the hope that the best solution will win.

There are also a few emerging services for doing Jingle-to-Phone calling: gtalk2voip seems to work well, I've also seen jabphone. gtalk2voip apparently now also support SIP interoperability (which is only described as currently free of charge).

And it seems like Asterisk Jingle support is on the way too...

In terms of open source support, Patches / Branches are available for Psi, Kopete and Gaim (although that one's a bit more tricky to get working). See my blog on building Psi and Kopete on Fedora Core 4. Neither were too complex, and that was a few months ago.

Unfortunately all of these patches/branches are languishing in we'll-finish-that-at-an-undetermined-date mode, as the projects are busy doing other things and so on. Yet they all seemed to work reasonably well, a lot of the remaining work is cleanup and merging to the main branch etc. (The one most likely to emerge in the official version is Kopete as its in 0.12, which is in Beta. But I'm not sure whether it'll be included in official builds on various distros...)

There are a number of proposals out there to do more work as part of Summer of Code, and I think it would be great if people signed up for these:

Jingle Audio Jingle Video Kopete Jingle Support

Zimbra is a fantastic open source mail server. It basically wraps Postfix, and has a beautiful webmail interface including a great calendar system. Yes, AJAX, buzzwords, all apply.

I've been wanting to migrate my mail into an IMAP server with a nice webmail frontend for a while. The advantages: being able to use mail from more than one machine, including my laptop when away and disconnected, other members of the family being able to use mail, etc, etc. Zimbra seemed like the best fit I'd seen since my last set of investigations (which incorporated dovecot, Hula, etc, etc).

At first I thought I'd just try out moving my mail in and see what problems I hit. By the time I had that done, I thought it would be a waste of time to retreat so I advanced headlong into the fray... Thus eating up a week of time or so, in which I was mostly unable to do very much with mail or anything else. (I should have thought beforehand, "I'm not really a mail admin person").

Anyhow, at the end of the process I think its been very successful and am very happy with Zimbra. This post is basically a summary of what I've learnt in the hope that other people will try out Zimbra and I will feel relief at the sense of not having to go through that again...

I had around 270000 messages in my Mozilla folder hierarchy, some dating back to around 1996. Total size about 4GB. This meant that the actual process of copying the mails was fairly cumbersome, especially if I wanted to check that everything copied OK (don't want to lose mail in the process!)

I installed Zimbra on Fedora Core 4, on my normal development / server machine. Don't be too concerned about the admonition to use a separate server, you can run Zimbra happily alongside existing instances of MySQL, Apache etc (Zimbra installs its own, hopefully in the future you can integrate it into your existing apps). I just needed to set Zimbra to only serve https, then I can access webmail etc on that port and my normal Apache takes care of port 80.

I then ended up developing some Python code to be able to do the things I wanted to fix up my mailbox etc. I've called this pyzimbra. These don't really consist of bindings to Zimbra code yet, but rather code to interact with the database, the message store, etc.

Issues encountered included:

Messages that refused to be copied (like finding a needle in a haystack), - I still have to resolve this one, only a dozen or so.

Dates on messages that didn't have proper date headers came out as 1st Jan 1970. Dates in the webmail view showed the time the message was copied rather than its original time. To fix these problems I wrote modules to connect to the zimbra database, and find the message file corresponding to each message in the database. I then checked if the message lacked a Date header and gratuitously added one based on other headers like Received (I'm sure this is naughty...). I then update the received date in the database (and alter the content-length if I added a date header). It was quite fun finding my way around the database etc, and the modules for doing this are fairly generic.

I'd also like to retrieve the Mozilla Flags (replied to etc) but unfortunately they don't get stored in the headers on the IMAP server after copying so that would involve finding the original message in the Mozilla mailbox that corresponds to the one in the Zimbra store.

Migrating my Mozilla mail filters to Zimbra: Zimbra uses Sieve for mail filtering. There is a nice utility for converting Mozilla's msgFilterRules.dat to a Sieve script (Javascript in a web page - use "/" as a separator in the option): mozilla2sieve. The Sieve script is stored in an LDAP attribute that I couldn't seem to access. You can access it via the zmprov utility, but there are caching problems.

The more serious issues are that Zimbra's Sieve implementation uses regular expressions, which makes havoc when you try and download a few thousand messages that have been queuing up and none of them get filtered - it will only display errors with the script when it actually tries to filter them. Look out for [ and *!!! Interestingly the Webmail Filters page basically deals with an XML representation of the Sieve script that it sends back and forth to the server.

If you're just running a few accounts through Zimbra and your main domain is hosted elsewhere, the Split Domain article on the wiki is really helpful.

Since I'm on an ADSL connection with a changing IP address, I'm using fetchmail to retrieve my mail periodically. Refused SPAM messages give a 501 response to the SMTP instruction.

Finally I wrote some nice backup scripts and utilities. zimbra/scripts/linkfolders.py is quite neat - it creates a directory structure on disk like the Folder structure in Zimbra, and symlinks the messages to their home on the message store. Nice for working out disk allocation, and as an alternate backup mechanism.

I've set up a copy of Zimbra, an open source messenging server with a beautiful AJAX webmail and calendar client on a local machine. Planned a while ago to move my mail to an IMAP server with a nice webmail frontend so it can be accessed from more than one place, and Zimbra seems like the best of the bunch.

The setup at first seems annoying because they expect a standalone server, and have their own copies of MySQL, Apache, PostFix, you name it ... Then I realised you can run all this alongside your normal software, as long as you set Apache to run on a different port. I guess it makes sense for them as a company that's really producing a commercial product (they sell a Network Edition which includes non-free Outlook connectors etc) and it could always be repackaged as desired.

I also found that ejabberd supports multiple distributed servers with failover which is pretty cool, and I might set that up myself...

Now it would be really nice if you could have distributed IMAP servers that automatically connected and synchronized in a similar way, but that's a fairly hard problem. Basically I'd want to run a local copy of the IMAP server on my laptop, my local server, and a master copy on the mail server, so that if any of them are disconnected from each other, mail is still available. OfflineIMAP might be enough though (although I can't honestly believe that its actually a Gopher site!).

Was excited around Christmas to see the Jingle signalling protocol for Voice over IP through Jabber, which was jointly authored by Google and JSF people and is used by Google Talk.

Google also released an open source library for doing Jingle called libjingle This makes it even easier for all the free software instance messaging clients to get the spec implemented quickly, and they all seem to have started doing that. Now that Google Talk has also turned on XMPP Federation, this looks like a blast for open protocols to win the day...

But when I looked at the roadmaps of Gaim and Psi, it became clear that although implementations are there, it'll take a while for these to reach general release.

Finally got round to trying to build some for myself, and found that they actually seem to work quite well (at least on my local network :-)).

Psi's jingle branch built fairly easily once I had the right dependencies etc.

I couldn't find instructions on building Gaim with Jingle support, it seems like they are still working on integrating it (in the Gaim-VV project), and I liked using Gaim because I can do IRC, Jabber etc all from the same client...

So I found Kopete which is similarly functional and has Jingle support in their development branch.

Building this as an rpm took more work because on Fedora at least, its part of kdenetwork which is fairly big. Eventually got it working after learning a fair bit about spec files... (rant: lots of "HOWTO" documentation, not enough reference documentation...)

The resulting RPMS, SRPMS and specfiles are all at http://davidf.sjsoft.com/files/jingle/ ... of course I should create a proper package repository but that can wait for another day...

Long time no blog ...

We're releasing a South African version of OpenOffice.org in all 11 languages. Some are still in Beta... This is based on milestone 128, and we actually did our own Windows build (no minor task!!!)

We'd like to get testing done on this release, so any volunteers are welcome!

This quote from the BBC's Have your Say: Microsoft after 30 years is really exactly the opposite of what we are about:

Microsoft has changed the world. At this point, the fact that they have such a large percentage of the market is a good thing. It has also guaranteed that English will be the language of the world for many generations to come. Steph, Decatur, Georgia USA

I hope in at most 10 years time, people's perception will be different

A few language related news items from South Africa over the last few months:

Firstly, the Pan South African Language Board decries the loss of human languages but only has International Mother Tongue Day and such like as solutions. Surely this is a feather-weight solution to a hippo-sized problem? For many people they'll pay as much attention as to International Talk Like a Pirate Day (or less). We need rather concerted efforts on the part of governments and civil society around the world.

For example, Mbeki raising the issue of language neglect with the National House of Traditional Leaders. This is great. Interestingly, he quotes the above report to say that in South Africa, only Afrikaans is defended by its speakers. Declining interest in African languages is reported in the Department of Education.

What can we do to fight these things? Promote the use of diverse languages in court, for a start - the article illustrates the kind of difficulties of encouraging multilingualism when there are several languages at stake. But it's a road that needs to be travelled. Fortunately the South African constitution mandates government promotion of languages. For translate.org.za's part, we're trying to ensure that people can use the language of their choice on their computers.

Behind all this is a mentality issue: are we as South Africans just going to imitate the west or are we going to celebrate our uniqueness and diversity? And keep a good sense of humour at the same time - sometimes localization goes awry.

Unrelatedly, I found this quote fascinating: Rebecca Wanjiku on the World Summit on the Information Society in Geneva, 2003:

Being my first time to attend intergovernmental negotiations, the chance was an eye opener for me. I witnessed "reputable" governments opposing press freedom, locking out major input from the civil society and at the end of it, I understood why my government never discloses its contribution to the public.

Hopefully South Africa will not allow the many winds and waves of global and local thinking to divert us from what is truly important.

It's really worth taking a look through the Free Software Magazine. Makes so much sense to have a magazine focussed on free software rather than some particular operating system. Now it would be nice to have regionally variant versions (say an African / Southern African one that adds locally relevant articles) - and that would be possible given the fact that articles are made available under the GNU Free Documentation License six weeks after publication.

Aaron J. Siego thinks that porting too much open source software to Windows will kill open source on the desktop.

The fear reminded me of a conversation I had with Dwayne and Alberto in Tanzania.

I said, I don't think we should define the platform in terms of the software. I like to think of the Free Software Platform - that's what matters to me. Obviously quality and design counts, but I wouldn't have that a problem running free software applications on top of something like ReactOS. So what counts is the freedom, the license, the philosophy - and what the applications do. What's underneath isn't as important.

Design a new splash screen for the upcoming 2.0 release - I like these ideas so far:

simple clean clean with a photo round buttoned splash bands classic mouette fantasy seascape dark vertical band

Funny how its easier to keep this list on my blog than in my bookmarks

Just leaving to go to a training camp for the Swahili Localization Project in Arusha, Tanzania ... should be fun but already missing Danielle and James and I haven't left yet!

Will be quite a comprehensive training camp and I'll be doing a session on using the translate tools for localizing Mozilla, OpenOffice.org, etc...