I've had a strange experience over the past year, and it's getting more common. As a developer, I used to be the one recommending software/tech things to my friends. Now all my non-technical friends have started recommending things to me that they have discovered on the intar-web. The trouble is, they're almost all closed systems - apparently free, but under the control of one group.

It started with Skype. Then xanga. And most recently it's been facebook. The speed with which facebook has spread amongst various of our friends is impressive. Some things about it are really good too - they're beginning to understand the sorts of things you can do with the Web.

But it's hard to explain that these sorts of things are the point of the Web as a Whole, rather than being a neat idea that can only work on a particular site. No-one understands the negative effects of a closed network. Or even more, the potential positive effects they're missing. Part of the problem is vocabulary - if you don't even have words to encapsulate the concepts to communicate, it's hard to argue for something. Open and closed networks are clear concepts to me but it's nice to embellish them somewhat.

So here's my attempt: A closed network utility under the control of one group that doesn't let you federate is called a nuct(a Network Utility Control Trap - don't let them innuct you into it). If you encounter one, you need a way of interacting with it that doesn't suck you in. The point of closed systems is of course, to prevent this.

But no-one seems to be doing the work required to break these things open We need a nuct-cracker suite(drum roll)

A simple start on easy ways to start breaking things open:

• open APIs to access data locked up in closed networks
  • a suite for different kinds of apps
  • social networks - same API to different ones
• plugins to open source open standards programs to interoperate with those networks
  • e.g. plugin to Gaim to control Skype
• open source servers with open APIs to replace the closed networks
• deploy and let the network effect take place

The trouble with tech pseudo-values: they're not the gospel. So on the one hand there comes a limit to avoiding things that would help good friendships because I think they could be done better. But more simply, I don't have enough time to actually create the online world the way I want it to be...

Is anyone doing something like this?

Zimbrais a fantastic open source mail server. It basically wraps Postfix, and has a beautiful webmail interface including a great calendar system. Yes, AJAX, buzzwords, all apply.

I've been wanting to migrate my mail into an IMAP server with a nice webmail frontend for a while. The advantages: being able to use mail from more than one machine, including my laptop when away and disconnected, other members of the family being able to use mail, etc, etc. Zimbra seemed like the best fit I'd seen since my last set of investigations (which incorporated dovecot, Hula, etc, etc).

At first I thought I'd just try out moving my mail in and see what problems I hit. By the time I had that done, I thought it would be a waste of time to retreat so I advanced headlong into the fray... Thus eating up a week of time or so, in which I was mostly unable to do very much with mail or anything else. (I should have thought beforehand, "I'm not really a mail admin person").

Anyhow, at the end of the process I think its been very successful and am very happy with Zimbra. This post is basically a summary of what I've learnt in the hope that other people will try out Zimbra and I will feel relief at the sense of not having to go through that again...

I had around 270000 messages in my Mozilla folder hierarchy, some dating back to around 1996. Total size about 4GB. This meant that the actual process of copying the mails was fairly cumbersome, especially if I wanted to check that everything copied OK (don't want to lose mail in the process!)

I installed Zimbra on Fedora Core 4, on my normal development / server machine. Don't be too concerned about the admonition to use a separate server, you can run Zimbra happily alongside existing instances of MySQL, Apache etc (Zimbra installs its own, hopefully in the future you can integrate it into your existing apps). I just needed to set Zimbra to only serve https, then I can access webmail etc on that port and my normal Apache takes care of port 80.

I then ended up developing some Python code to be able to do the things I wanted to fix up my mailbox etc. I've called this pyzimbra. These don't really consist of bindings to Zimbra code yet, but rather code to interact with the database, the message store, etc.

Issues encountered included:

Messages that refused to be copied(like finding a needle in a haystack), - I still have to resolve this one, only a dozen or so.

Dates on messages that didn't have proper date headers came out as 1st Jan 1970. Dates in the webmail view showed the time the message was copied rather than its original time. To fix these problems I wrote modules to connect to the zimbra database, and find the message file corresponding to each message in the database. I then checked if the message lacked a Date header and gratuitously added one based on other headers like Received (I'm sure this is naughty...). I then update the received date in the database (and alter the content-length if I added a date header). It was quite fun finding my way around the database etc, and the modules for doing this are fairly generic.

I'd also like to retrieve the Mozilla Flags (replied to etc) but unfortunately they don't get stored in the headers on the IMAP server after copying so that would involve finding the original message in the Mozilla mailbox that corresponds to the one in the Zimbra store.

Migrating my Mozilla mail filters to Zimbra: Zimbra uses Sievefor mail filtering. There is a nice utility for converting Mozilla's msgFilterRules.dat to a Sieve script (Javascript in a web page - use "/" as a separator in the option): mozilla2sieve. The Sieve script is stored in an LDAP attribute that I couldn't seem to access. You can access it via the zmprov utility, but there are caching problems.

The more serious issues are that Zimbra's Sieve implementation uses regular expressions, which makes havoc when you try and download a few thousand messages that have been queuing up and none of them get filtered - it will only display errors with the script when it actually tries to filter them. Look out for [ and *!!! Interestingly the Webmail Filters page basically deals with an XML representation of the Sieve script that it sends back and forth to the server.

If you're just running a few accounts through Zimbra and your main domain is hosted elsewhere, the Split Domainarticle on the wiki is really helpful.

Since I'm on an ADSL connection with a changing IP address, I'm using fetchmail to retrieve my mail periodically. Refused SPAM messages give a 501 response to the SMTP instruction.

Finally I wrote some nice backup scripts and utilities. zimbra/scripts/linkfolders.py is quite neat - it creates a directory structure on disk like the Folder structure in Zimbra, and symlinks the messages to their home on the message store. Nice for working out disk allocation, and as an alternate backup mechanism.