Mother Tongue Bloggers

Virtaal supports Haitian Creole through Machine Translation plugin

2010-02-09T13:35:12+00:00

Update: Originally published on 2006-01-26 but somehow I didn't get this pushed on to the RSS feed so I've published it with a newer date.

Virtaal, a Computer Assisted Translation (CAT) tool, has been providing translators with Machine Translation suggestions through its plugin system. We've just committed a new Machine Translation plugin that allows Virtaal to use Microsoft Translator's new Haitian Creole translation engine.

The Microsoft Translator plugin has been waiting for the next major release as we didn't want to introduce any User Interface changes. But the usefulness of this tool in the current Haitian crisis means that we've released it so that people can use it and benefit from the feature.

How does this help?

Any software that needs to be localised into Haitian Creole can now be more easily translated in Virtaal with Haitian Machine Translation support.
Documents (OpenDocument Format and wiki texts) can be translated into Haitian Creole using Virtaal and the Translate Toolkit's txt2po and odf2xliff converters.

How do I use the plugin?

You have two options:

Windows: Download our special Virtaal .exe for Windows.
Linux: Copy the Microsoft Translator plugin into your plugin directory (on Linux this is /usr/lib/python2.6/site-packages/virtaal/plugins/tm/models/

How to use the plugin?

You'll need to define Haitian Creole as a language in Virtaal

Bottom Right > New Language Pair...

New Language Pair..." /> New Language Pair..." /> New Language Pair..." /> New Language Pair...">

Add Missing Language...

Name 'Haitian Creole' and language code 'ht'

Ensure that the plugin is enabled. Edit > Preferences > Plug-ins > Translation Memory > Configure... Ensure Microsoft Translator is checked.

Open your file and begin translating, I'm using the tutorial Help > Tutorial and as you can see I'm getting Machine Translation suggestions from Microsoft for Haitian Creole.

Translate Toolkit - a powerful localisation toolkit

2010-02-04T06:49:35+00:00

What did it take to allow Pootle, our web-based localisation platform to support Qt Linguist (.ts), TMX and TBX formats?

Three lines of code!

Pootle uses the Translate Toolkit which Translate.org.za has developed since 2002 to provide a powerful set of tools to manipulate localisation files. The Toolkit has evolved and grown as our needs have changed and as we began to localise other applications like Mozilla Firefox and OpenOffice.org.

The toolkit has the concept of a localisation base class from which all our localisation file implementations derive. We use the base class within Pootle to access XLIFF and PO files. We were pretty sure we'd be able to support other bilingual files (localisation files with both the source and target language in one file) in Pootle but we didn't think it would be this easy.

Of course we'd like to make some further changes so that Pootle simply supports any bilingual file store from the Translate Toolkit and we'll need to test Pootle running with this new feature, expect a beta soon.

Now you might be wondering about monolingual files (localisation files with just one language stored in the file e.g. Java properties files). Well we're busy looking at those and hopefully with the next release of Pootle we'll be able to directly translate monolingual files. Have a look at our list of currently supported localisation files and start dreaming about what will be possible.

For software and content developers the exciting new is this. You want to localise your esoteric format in Pootle? Simply write a storage class for the Translate Toolkit and you will automatically have support within Pootle for your format.

Update: Just before publishing this blog post the Translate Toolkit development list got news of a new Gettext PO diffing tool that uses... the Translate Toolkit of course.

The sky's the limit for new Zulu spell checker

2010-01-28T21:02:04+00:00

Translate.org.za are the proud parents of a new Zulu spell checker.

What makes us such proud parents? We've ported the spell checker from the Myspell platform to Hunspell. Which means what exactly? It means that we can now spell check Zulu text at much higher precision. It also puts the platform in place to ratchet up the checkers performance.

You can try the checker as an extension for OpenOffice.org, Mozilla Firefox or Mozilla Thunderbird.

Spell checking in Zulu is hard because it is a highly conjunctive language. This means that what in English would be seen as a number of separate words is written as one word. What this means for the spell checker is that we first need to deconstruct the Zulu word. We then check that the remaining root is a correct word and that the rules for building the expression where correct. If we fail any of that then we have a spelling error.

We have most of the most productive classes of nouns and verbs covered. Our work now is to expand the rules as needed and most importantly to add and classify many root forms.

The initial work for the Zulu spell checker was done as part of the ANLoc spell checker project with funding from the International Development Research Centre.

It seems my web browser is unique

2010-01-28T13:35:42+00:00

I read yesterday about Panopticlick which tries to determine how easily web users can be traced without the use of web cookies. It collects information sent by the browser in the HTTP protocol, and things that can be collected by means of JavaScript, Flash and Java. The website reports that my web browser is unique in the pool of browsers that visited Panopticlick.

The interesting/worrying thing is that most of the fields could identify uniquely me on their own. Nobody else uses the same language combination for HTTP's Accept-Lang (the list of languages that I prefer). Nobody else uses the same version of Firefox in Afrikaans on my Linux distribution.

I won't go into why this is good or bad, but it is interesting to know that if I had reason to not want to be identified, I can't use Firefox in Afrikaans any more, and I can't indicate my preferred language to web sites. I realise that plugins can probably improve the matter, but if this is seen as a danger, it is probably a danger for more people than the number of people who would know about NoScript or TorButton.

Should we perhaps think afresh about how much information is sent by web browsers? At least I now feel a little bit more special...

Everyone has the power to champion their language

2010-01-26T09:33:40+00:00

Mondli Makhanya, editor of the Sunday Times (South Africa), raises an interesting point about Afrikaners and language in his piece, "Afrikaners set a fine example in championing their language".

While Mondli places much of the blame on government, and I would agree that there is much blame that can be placed on government's shoulders, is it not the speakers of a language who carry the greatest responsibility? While the Afrikaners are concerned about this issue where are the Zulus, Vendas and Tswanas?

The biggest problem with our constitution is that in appointing PanSALB as the custodian of our languages we seem to think that that means we don't need to do anything!

When an Afrikaans school is forced to become dual medium because black parents want English education for their children is that not a concern? It's a grave concern that most people see this as a conservative white issue. While I'm sure it had those elements, isn't it of more concern that non-English speaking children are getting a bad education by being forced to study in English?

Mondli talked about UNESCO's monitoring of the situation of dying languages. What he doesn't mention is that UNESCO champions mother language education because time after time it has been shown that mother language education is better. It leads to more engaged students, better thinking and better assimilation of fundamental concepts. All of these seem to be missing from the poorest of our schools who hack along in English. So it is a grave concern to me that we are forever assigning children to inferior education because we hope that English education will make them the best.

Where were the mother tongue speakers when UNISA closed down much of its African language department? Not a peep.

When I go to home affairs and see only English forms why am I the only one to complains (English is my mother language)?

The beauty of seeing language as a personal issue is that then you are able to change the situation. If you take language personally then you will decide to speak your mother language at home. If you take it personally then you will use the ATM in your language, you'll tweet and post status updates on Facebook in your language. You'll set your cellphone to your language.

Lastly, you'll claim your workplace for your language by insisting that you use computer software in your language. Organisations like Translate.org.za have been making mother language technology such as the Venda keyboards, Firefox spell checkers, South African calenders and other technologies for years and you can empower your language by simply using them.

When we blame government we remain powerless, when we make language our priority we can change the world.

Pootle empowers translation teams

2009-12-15T14:21:48+00:00

With the recent release of Pootle 2.0 I want to write a bit about some of the nice features that Pootle offers. Pootle empowers translation teams to more easily do their work, and to improve quality. Where there might be differing skill levels (for example with crowd-sourcing), these functions are of crucial importance.

Pootle makes it possible to configure permissions that determines who is allowed to do which tasks in a certain language. There are settings that apply to the whole server, and the language team's settings can be further customised for a specific project where necessary, but the most powerful configuration is probably where most things are configured on the language level. It is easy to configure that some people may only make suggestions, while others are allowed to edit translations directly.

The pages with news can help a team to stay up to date with all the activities. Team leaders can simply write news that will also be available in the RSS feed, but Pootle will also contribute by generating notices that informs users about files that were uploaded, new projects, when a file reaches 100% status, etc.

Furthermore Pootle helps with terminology by showing suggestions from a terminology project during translation. This helps with consistency, especially if newcomers are contributing. There is also a big selection of quality checks available that helps to find a variety of problems.

But Pootle tries to not impose any restrictions on a team. People who want to translate offline can easily download files and complete their work with something like Virtaal - even without an internet connection. After files are uploaded, all the other functions can still be used. Team members can even work directly with version control, and Pootle can then integrate with such a version control system.

What I'm loving about Virtaal 0.5.0

2009-12-09T07:36:45+00:00

With the release of Virtaal 0.5.0 I've had a little bit of time to reflect on this tool.

Virtaal is a CAT (Computer Assisted Translation) tool, a tool designed to help human translators translate more effectively. From the comments we get from various localisers they're loving Virtaal, they're working much more quickly then they have on any other tool.

For me Virtaal is all about power and simplicity. Lets provide the most powerful features to the translator while allowing them to focus fully on the task of translation. We don't ask them to do massive configuration before they can even translate, rather we wait to ask them when we really need that piece of information.

Here are some of the new features that I'm loving in Virtaal 0.5.0:

Autoterm provides terminology files for Virtaal to download and use. You didn't know there was an Urdu localisation terminology list? Out hope is that this puts good terminology into localisers hands.
Web look-up allows you to look-up some selected text on websites. Tired of copying text, launching your web-browser and querying? Then simply select the text, right click and choose what search engine to use.
Highlighting of differences in non-100% TM matches.
New plugins for machine translation with Google and Moses. We've added these two on top of the existing Apertium machine translation.
Language identification for easier selection of language pairs. For the inner geek in me. We will now identify the source and target text if the file format doesn't tell us. Its helpful for someone like me that is often changing between various target languages.

That's just a scratching of the surface. I hope to give some more details in future posts.

Pootle 2.0.0 released

2009-12-08T10:05:22+00:00

Yesterday the Translate team released version 2.0.0 of Pootle. Pootle is web based system for translation and translation management. It is widely used for crowd-sourcing, especially in the world of Free Software.

This is a major new release after a long period of development. It has many new features, and Pootle is now built on a completely new web platform (Django).

This work was made possible by many volunteers and our funders:

ANLoc, funded by IDRC
Mozilla Corporation

Thank you to everyone involved!

To see Pootle in action, or for further translation updates, go to http://pootle.locamotion.org/

Highlighted improvements

A new user interface
- more useful information
- a pretty new design
- easier customisation of the Pootle design
- improvements to alternative source languages
- news feeds on several levels of Pootle
- automatic notices for certain events on Pootle
- better account and password management
Easier management from the web interface
- uploading template files
- removing files
- scanning the file system for translation files
- removing languages from a project
- the possibility to upload all translations as suggestions
- server permissions
- language permissions
- delegation of administration rights (per language and server wide)
- language administrators can manage all aspects of their language
Better performance
- many pages are served from cache for anonymous users
- support for more concurrent users
- support for larger deployments
- faster uploads
Easier deployment
- can run from a checkout / extracted archive without installing
- interface translations from the Pootle project or .mo files
- performance tips on the admin page
- creating initial database content if needed
Back-end improvements
- built on Django
- can now use many database systems, including MySQL
- can run standalone or under Apache
- can use memcached
- LDAP support
- migration script for older Pootle versions

Older Pootle installations can easily migrate everything excluding project permissions. We encourage administrators to configure permissions with the new permission system which is much simpler to use, since permissions on the language level are now supported. Goals and assignments will receive attention in a future release. Read more about migrating from older versions, or contact us on the mailing list if you have any questions.

This release will also benefit from improvements made to the Translate Toolkit since the last Pootle release, such as speed improvements, better XLIFF support, and better quality checks.

Virtaal 0.5.0 released

2009-12-04T11:27:43+00:00

Yesterday the Translate team released version 0.5.0 of Virtaal. Virtaal is a translation program with powerful features to help translators increase translation quality and have fun. It is available for download from SourceForge and soon your favourite Linux distro.

This release was meant to only add a few small features, but ended up with many other enhancements and nice bugfixes.

Highlighted improvements

Autoterm provides terminology files for Virtaal to download and use (even if the internet connection disappears later)
Web look-up allows you to look-up some selected text on websites
Highlighting of differences in non-100% TM matches
New plugins for machine translation with Google and Moses
Correctly support Open-Tran.eu for languages with regional modifiers
Web requests now accept compressed responses if the server offers it
Language identification for easier selection of language pairs
New autocorrect files for Luxembourgish and Croatian
Updated translations as well as these new ones: Sotho, Northern Sotho, Songhay, Korean, Ukrainian. Contact us to add yours!
Several bug fixes

Users of input methods such as ibus should see their issues fixed, but need GTK+ 2.18 (on Linux). Many thanks to Sankarshan Mukhopadhyay, Peng Huang, and Matthias Clasen for helping to resolve these issues. (bugs 1119, 1281)
This release will also benefit from some improvements in new releases of the Translate Toolkit, including support for OmegaT glossary files, and the experimental new PO parser which is much faster.

The Virtaal user manual is an excellent way to learn how Virtaal can help you to be productive.

Interview for the SourceForge community blog

2009-11-26T10:26:53+00:00

I was recently interviewed about the Translate Toolkit. It was published on the SourceForge community blog. Apart from the useful tools that form part of the Translate Toolkit, it is of course also a fantastic platform / API for the development of new programs that have to do with translation, so I'm quite happy about the exposure there. Read the article here.

Translation Memory difference highlighting in Virtaal

2009-11-15T13:57:01+00:00

Translation Memory, everything that you've translated in the past, is an amazing resource. But like all technology to assist a translator it is only useful if it is quick and easy to use.

Virtaal has been growing an impressive array of Translation Memory plugins (also some pretty cool Machine Translation plugins), this means that you are potentially getting more an more matches. The problem with our current implementation is that it's hard to know exactly what the difference are between your current segment and the segment matched by TM. Is it a word, a spelling difference, some punctuation?

With the upcoming release of Virtaal 0.5 we'll see that change. Over the weekend I implemented difference highlighting. Instead of explaining what I mean let me just show you in the following picture.

As you can see from the screenshot above both suggestions provided by Virtaal differ from the current source segment. The first has a space at the end. While the second one is capitalised differently. It's easy now to see how these suggestions are different and easy then to choose which one to use and where to edit it so as to bring the suggestion in line with what is needed for the current segment. Now imagine this with a whole sentence.

Now that's a bit easier then trying to work out how those two suggestions are different from the current segment. As a localiser or translator that's about all you need to know.

Now for the technically inclined. I'm using difflib to implement the differences. Its quite powerful and pretty easy to use. The get_opcodes method provides a nice way of determining what has changed. I then take the segment and wrap it in Pango formatting instructions to arrive at the nicely rendered output.

The implementation is not without issues. Difflib does its job too well. When you see a change of one word for another then to a human you want to see the one word deleted and the other inserted. But to difflib things seem very different, it looks at the character level and seems to find patterns of characters, trying very hard to show you how the one word changed to the other. It is simply impossible for a human to read. So we'll need to look at that and work out some way to adjust the get_opcodes output turn that complication into a simple replace of the whole word.

There are other places where we want to put difference highlighting. The previous msgid functionality of PO files is one, there will be others. But that's going to happen in the Virtaal 0.5 timeframe I'm afraid.

Sneak peek at the new Pootle

2009-11-11T14:08:57+00:00

The Translate project has been working very hard on the upcoming release of Pootle for the last year, and things are looking great for the upcoming release. We recently released beta-4, and things are getting close to release quality. Kudos to Alaa and Wynand who did most of the work. We also had contributions from several other companies and individuals. There are loads of improvements since version 1.2, but I wanted to write specifically about some improvements to the user interface that translators and localisation managers will enjoy.

Yes, the rumour is true: we got rid of the Pootle mascot (known to some as the potato man). He served us loyally for a few years, but we are happy to see him go as we say goodbye to the days of the old Pootle.

In fact, there is a completely new look. Have a sneak peak on our testing server. Many people have worked on making this happen, including people from Mozilla, Manal Hassan and Julen Ruiz Aizpuru. In fact, Julen has now become the most active contributor to our user interface. I think it looks much more modern and pretty, and presents a lot of functionality much better.

The front page now has much more information. It actually gives a complete overview of things happening on the server. This idea came from the contributors at Mozilla. It also provides a summary of the top contributors on the server, and similar summaries are displayed on several sub-pages of the server.

Administrators will find far more of the configuration options in the web interface. Previously some options could only be found in the configuration files. It will also be easier to find some of the admin pages from the projects that they relate to.

A feature that I'm very excited about, is a better organisation of the activities in the project view. Pootle's quality checks are loved by all its users, but it has been a bit hidden. Things are now separated into divisions that will make more sense for the users.

All of this promises a very exciting release of Pootle. Join us now in the final stretch to make it happen.

Wikipedia now with more articles in Swahili than Afrikaans

2009-10-29T07:37:43+00:00

I noticed recently that Wikipedia in Swahili has more articles than Wikipedia in Afrikaans. The difference is already around 9%. I hope and believe that most people know that the article count at Wikipedia doesn't say everything about the relevant encyclopaedia, but it remains an interesting toy, and it is interesting to see how things develop among the African languages.

I have been keeping an eye on the Wikipedia project for some time. It remains a wonderful demonstration of the power of a strong community. It is also a way for a number of minority languages to create a useful web resource, and for many languages, perhaps even their first encyclopaedia. The Afrikaans version of Wikipedia was the largest in terms of article count for a long while, but now Swahili is ahead. It must have happened in the last few months. Congratulations to all the contributors to the Swahili version!

Tables with interesting summaries of the statistics are available for Afrikaans and Swahili. (You might want to compare with the English version to understand the table.) Of course it is important to note that the two languages differ quite a bit in other aspects of growth and size. Another interesting page for the purpose of comparison is the list of all Wikipedias. More detail about article sizes is also available for Afrikaans and Swahili. If I look at this, I would like to believe that the Afrikaans Wikipedia still has an edge in a few areas, especially in terms of article size and shear amount of text. However, it does seem as if the Afrikaans version's growth isn't really accelerating.

Luckily this is not a competition with losers, and I'm very happy for good progress in any African language.

FIrefox in 10 African language

2009-10-21T13:43:09+00:00

Yes 10 African languages for Firefox. Officially there is 1 (one) African language (Afrikaans) in Firefox, not counting languages of wider communication like English, French, Portuguese and Arabic. ANLoc (The African Network for Localisation) is trying to change that literally from A-Z, Akan to Zulu. With of course Northern Sotho, Shona, Krio, Wolof, Songhai, Swahili, Lingala and Luganda in between.

ANLoc is working through the following process to try to create the skills and support needed to get more Africans localising Firefox:

We asked Africans to submit a proposal on why we should support their language team. We selected the 10 best proposals from those that we received.
A tech team was created with people skilled in development and localisation. The idea is for this group to support less skilled localisation teams and also to grow the skills within this group. The tech team is spread across Africa. They are doing things like managing files, building language packs, helping give input to localisers and in the longer term helping the new teams interface with Mozilla so that they can grow their own wings and fly.
Hosting the teams on Pootle a web server for managing software localisation. We've put all the localisation files on Pootle to help track progress and assist teams that are less technical. Those with bad connectivity, a common issue in Africa, can download the files, translate offline and then submit their work through Pootle.
Working with the teams to help promote their work in the own country.

Why Firefox? It's a widely known product, it's used by end-users (the people who need local language) and it's a product produced by an organisation that is passionate about championing the open web. Mozilla has done more for the WWW then most, ANLoc is standing on the shoulders of giants. Thank you Mozilla.

How can I help? Here are a few ideas:

Download and test a language pack, especially if you speak one of the 10 languages.
Speak French? You can really help some of the localisers who are from French speaking Africa trying to localise into their own language
Tell others about this effort, especially if you have some connection to the language listed. You are in the diaspora, you speak the langauge, you come form a country that speaks the language.

ANLoc is a network of passionate language experts funded by the International Development Research Centre.

Ubuntu PPA for Translate Toolkit and Virtaal

2009-10-19T09:58:03+00:00

I have recently created an Ubuntu PPA to make the Translate.org.za software more accessible. At the moment only the Translate Toolkit and Virtaal is packaged, with Pootle to follow soon.

Please let me know if there are any questions, problems or comments.

Pseudolocalisation with podebug (3): Interview with Rail Aliev

2009-09-18T13:35:24+00:00

This is part of my series on podebug. Last time we looked at identifying strings from different locations. This time I decided to conduct an interview with Rail Aliev who has been a major user of and contributor to podebug. I was specifically interested in his use of the hashing functionality of podebug, but hopefully you will also get to know this major contributor to the world of Free and Open Source Software.

Please tell us a little bit about yourself and your involvement in the world of Free Software.

I'm mostly involved in localization efforts (Russian and Turkish) in OpenOffice.org and Mozilla projects. Zemberek linguistic project (Turkish) is another interesting one. As result I maintain some linguistic packages in Debian.

And the last but not least, the Translate project.

Please tell us about your preferred translation environment and how things work in your teams.

Currently I use the following tools:

translate-toolkit for format conversion (oo2po, oo2moz), testing translations (punctuation, spacing, expressions, etc), debugging (podebug) and other off-line helpers
Pootle for collaborative translation, quality assurance (web interface for translate-toolkit's features), translation repository
Virtaal for translation. Virtaal provides great features, such as on-line translation memory queries from OpenTran project, automatic translation using libtranslate. And all these features in version 0.4! Of course, there are a lot of features to be implemented, but, as I said, this is a good start.

Tell us a bit about how you use podebug, and specifically the –hash option. Why do you find it useful?

I use podebug only in cases when you cannot find the string, which was found as a wrong translation.

As a generic example, if you have English “Right” string, you can translate it as “right side” or as “not wrong” (in real world, depending on a language, the list of translations may be extended). Even worse, sometimes you have this string used more than once in a file (it's very common in OpenOffice.org project, where files with thousands words aren't rare). It's the case where we can use podebug's power.

As an example, here are the “magic” commands to get debug version of SDF file, used in OpenOffice.org translation:

Create normal PO:
oo2po -l tr en-US_tr.sdf po
(po folder will be created)
Create debug PO:
podebug -f "%h." po po.debug
Create debug SDF file:
po2oo -l tr -t en-US_tr.sdf po.debug tr-debug.sdf

The most painful steps are first and latest. You need OpenOffice.org build tree and you need to know how to build it.
The OpenOffice.org interface becomes something like this (click for the full version):

Now you can definitely find the string you are looking for, first in po.debug folder and then fix the corresponding string within po folder.

Keeping po folder under version control saves your time when you are ready to merge your fixes upstream.

Any other tools/options that you use to compliment podebug –hash?

You can play around -f option of podebug to get a different prefix for messages. For example, -f "%2h: " will add first 2 letters of the hash, semicolon, space and the translated message. The hash is computed using string's comments and context MD5 checksum, so it's stay the same even if English source text is changed.

Showing the current git branch in the Bash prompt

2009-09-17T09:25:45+00:00

This is just a simple explanation of how I hacked my BASH prompt to show me the current Git branch of my current working directory.

Having done one too many git svn rebase commands while on the wrong Git branch, I started looking for a way to always know which branch I'm working on. What better place to do this than hacking it into my BASH prompt?

Lo and behold, there in the PROMPTING section of the bash(1) man page (or this page) is my answer: "...After the string is decoded, it is expanded via parameter expansion, command substitution, arithmetic expansion, and quote removal...".

From my ~/.bashrc:
PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u\[\033[01;37m\]@\[\033[00;30;43m\]\h\[\033[00m\] \$ ' export PS1="\e[1;37m[\w]\e[1;34m\`$HOME/scripts/prompt_add.sh\`\e[00m\n$PS1"

The bold bit is the only important bit for the issue at hand. It simply replaces the bold text with the output of the script it runs. A very important thing to note is that the back ticks are escaped (thanks to Joshua Price's blog post about BASH prompt hacking for showing how this should be handled). Without escaping the back ticks, the prompt_add.sh is only run once when ~/.bashrc is read (login time).

All that's left is the prompt_add.sh script itself:
#!/bin/bash

if [ -n "$(git status 2>&1 | grep -v '^fatal: Not a git repository')" ]; then
echo \[git: $(git branch | grep '\*' | awk '{ print $2 }')\]
fi

Probably not the nicest way to determine the Git branch, but it works.

Come to thing of it, you can add any code to that script that you would want in your prompt. Hmmm... what else can I jam in there? >:)

Where is the modularity?

2009-09-03T14:24:48+00:00

Yesterday I saw that there are some updates for my computer. On closer investigation, the issue is summerised thus:

Importance: bugfix
Reason to upgrade:krandrtray from KDE4 is known to have some issues.
A patch was added that makes krandrtray open its configuration module when the system tray icon is clicked.

All nice and dandy. I don't really use KDE any more, but like to take a look every now and then. The "interesting "part is that I have to update 28 packages. Summary:

16KB of additional disk space will be used.
50MB of packages will be retrieved.

Now why on earth 28 packages for a bug in krandrtray?

Support for Afrikaans at Google Translate

2009-08-26T12:14:25+00:00

Google is constantly extending its software for automatic machine translation. A new set of languages that was recently added, includes Afrikaans and Swahili. This is the first two languages from Africa that is added to the list. This is interesting for several reasons, and I'm wondering how this can change the landscape for these two languages.

As an Afrikaans speaker, the first reason why this is interesting, is to see how well it fares, and to see which mistakes it makes (totally expected, of course). We all know that Google uses statistical machine translation. This theoretically means that it should just keep on improving as they continue to get more data to work with.

Interesting mistakes that I noticed in translation from English to Afrikaans; (please excuse possible mistakes in grammatical terms. Afrikaans ones added for reference.)

Morphology (woordbou). Compounds are mostly not handled correctly. It knows about something like "Wêreldbeker" (World cup), but probably just because it encountered it before. How well does statistical machine translation handle target languages like German and Dutch? Will more data make the problem go away?
Words with apparent Dutch or German inspiration, such as "epigrammatisch", "gewijd", "gefascineerd" that I can't imagine coming out of any Afrikaans source.
The article 'n is is frequently wrong. It occurs very often as vir' n been (with the apostrophe stuck onto the previous word. It seems that the apostrophe is handled as a quotation, and then it closes the "quotation" from time to time with the apostrophe.
Where sentences start with the article (lidwoord) 'n, the use of capitalisation is wrong.

Dwayne mentioned that some of the mistakes could be due to training from texts that were collected through optical character recognition. This would explain the problem with the apostrophe, for example. Although statistical machine translation might be language agnostic, the same is definitely not true for optical character recognition.

An interesting one to see was the translation ">">"lang-en korttermyn-verhoudings" for "long and short-term relationships" — not a bad attempt. The mistake with the incorrect "distant compounding" (afstandsamestelling) can easily be due to optical character recognition.

A few more comments about this:

African languages is important enough for Google to put this effort in. Although it is later than what we would have liked, at least this is something. I hope more companies will take note and follow the example. It is interesting to see that Afrikaans is supported before some big languages of India. (With big I mean a language like Bengali with more than 200 million speakers.)
Google is not first. The Apertium-project has had a translator for some time already, which works on totally different principles (it is rule based). I would recommend any interested people to collaborate with the Apertium project to improve their software, especially for languages with fewer resources. They are good at helping people who want to contribute. You don't need to have programming knowledge.
It works, sort of. Somebody who doesn't understand Afrikaans, should be able to to get an idea of what is written in an Afrikaans text. I did however not get the idea that you would be getting a good idea with the current quality of translation. Try it out and comment below. Humorous examples are especially welcome.
Due to the previous point, I believe Afrikaans people can now start to write more in Afrikaans, especially if the intended audience is partially Afrikaans. An argument for making a weblog more accessible for a theoretical international audience doesn't measure as strong as before. I translated this blog post myself. Can I use automatic translation from now on for the English version of my weblog? How accessible will it be?
I have no idea how well it translates into languages other than English. I also haven't yet tested it with source languages other than English. I'm keen to hear if somebody can evaluate this.
Now more than ever we will need to inform people about the limitations of machine translation. It will be a huge insult if people start to use this without realising that it can in no way and under no circumstances serve as a substitute for a professional translator. There will definitely be people who will want to use this incorrectly, even if they mean well by doing so. If it can't manage the indefinite article 'n, why should we trust it with our marketing material?
It doesn't matter all that much how good or bad this is. Because it caries the Google name, and will be integrated with other Google services, it will probably become the machine translation system that people will use.

Firefox in African languages: Who is joining us?

2009-08-24T11:35:20+00:00

As part of the African Network for Localisation (ANLoc) a project is being launched where several teams will collaborate to translate software into a few African languages. This is an opportunity to translate Firefox into new African languages. ANLoc will help with the process. More information available here:

http://www.africanlocalisation.net/call-applications-language-teams

Pseudolocalisation with podebug (2)

2009-07-31T09:14:41+00:00

A while ago I started with a series on pseudolocalisation in podebug. I showed how one can test translatability of a program with the xxx rule, and the final picture showed what it looks like when a string from another file is involved. Let's have a look at a way that translations from different files can be recognised more easily.

It is common for software to use a number of libraries, and therefore also the translations of these libraries. In the case of GNOME programmes (that use GTK+) a few common strings are often used from the translations of GTK+ for the most common things that repeat. For example: Copy, Paste, Preferences, etc. Therefore these strings are usually not translated in the programme's translation file itself, but in the file for GTK+. For translations of OpenOffice.org and of Mozilla programs, the translations are spread over several files, which can sometimes make it harder to know which strings came from which files.

podebug --format="[%7b]" virtaal.pot fr.po

With this command podebug creates a PO file based on the POT file, and "translates" it so that it looks like this:

#: ../share/virtaal/virtaal.glade.h:8

msgid "General"

msgstr "[virtaal]General"

In the program, it then looks like this:

We ask that podebug prepend a formatted string to the translation with the first 7 characters of the filename from which the translation comes. In this screen shot we see that all the translations come from Virtaal's file, except for "About" which comes from GTK+. There are several possibilities for the format string to represent the file or directory name. Read more on podebug's web page.

Here is a more complex example where a few files were annotated with podebug:

Here we see that the entries on the context menu come from GTK+, that the string "Fuzzy" comes from Virtaal, that the language name comes from ISO 639, and that the country name comes from ISO 3166. Of course I refer to the translations from the iso-codes package maintained by Christian Perrier of Debian fame. So here we see translations that are brought together from four files.

So this provides a way to see how translations from different files are combined.

Firefox 3.5 released

2009-07-01T09:11:59+00:00

Many people might have heard already, but I'm quickly writing anyway that Firefox 3.5 was released yesterday. When I looked just now (2009-07-01 11:00) it was already downloaded more than 6 500 times from inside South Africa.

In terms of the Afrikaans version:

All the new text is translated (thanks to Samuel Murray)
A few small errors were corrected (thanks to Dwayne and Johan Cronjé)
A few of the Alt+letter combinations were changed in the main menu
The new product pages were translated (thanks to Samuel)

Spread the news so that we can have thousands of South Africans switching to Firefox 3.5.

Presentation at Afrilex-Alasa 2009

2009-06-30T07:08:20+00:00

Next week I am giving a presentation at the Afrilex Conference at the university of the Western Cape. The title of my presentation is "Will translators use your dictionaries?" and will discuss how terminology can be developed with translation and optimal integration with tools for computer aided translation in mind. I will also give a demonstration of some of our translation programs. Let me know if I should look out for you or if somebody wants to meet up during that time.

Pseudolocalisation with podebug (1)

2009-06-16T09:58:47+00:00

The Translate Toolkit has had a program to help with pseudolocalisation since 2004: podebug. This is the first in a series of articles about podebug and what it can be used for.

Pseudolocalisation is a way to quickly generate or manipulate translation files to use for testing. This way the translatability or (internationalisability) of a program can be tested without having to translate it first or to review it on the level of the source files. It can also help translators in that translations can be annotated and can therefore be found more easily.

One of the things for which our team uses it frequently in the development of Virtaal, is to check if all strings are marked for translation.

podebug --rewrite=xxx virtaal.pot fr.po

With this command podebug creates a PO file based on the POT file and "translates" it so that it looks as folllows:

#: ../share/virtaal/virtaal.glade.h:8

msgid "General"

msgstr "xxxGeneralxxx"

In the program, it looks like this:

Now one can see that all the entries are "translated", except for "About". This can indicate that a string is perhaps not marked for translation. In other cases (as is the case here) it indicates that the string is translated elsewhere. This string is part of GTK+, and the translation will be retrieved by GTK+ from another file.

So this is a quick and easy way to check if all strings are marked for translation.

Google Chrome uses Hunspell

2009-06-05T09:44:52+00:00

I recently learnt that Google Chrome can also check spelling in text areas. It made me think back on how things changed in this area in the past few years.

If I remember correctly, I switched to Konqueror in ancient days since it was the only browser that could do this for me at that stage. I spent a lot of time at Wikipedia and needed an Afrikaans spell checker. This was also the time that I got really interested in spell checkers and I became more involved at Translate.org.za.

At some stage it became possible to also use the SpellBound extension in the then new Firefox to check spelling, and since version 2, Firefox could do it itself if I remember correctly. It is interesting to see that SpellBound is still available for people who want to check a lot of text at a time.

Apparently Opera can use Aspell, but it doesn't work on my computer, and I have heard that people struggle with it. Aspell also doesn't provide advanced technology that many languages need, and these days, most languages use Hunspell.

Of course on Mac OSX there is the system wide spell service that all programs can use. This is the way things should work, and as I understand, this is also the case in KDE4. This is definitely on my wish list for GNOME 3. At least Gtkspell already moved to Enchant which makes it possible to use Hunspell and other spell checking systems. For those who want to use Hunspell as the spell checker on OSX, have a look at OpenXSpell. This project holds a lot of promise.

It was interesting to see that Google's browser Chrome can also check spelling, and that it uses Hunspell directly. You can read their instructions and ask for the inclusion of any missing dictionaries (currently they don't package all the available dictionaries). Ain Vagula says his request was answered in a few days. Olivier R added that there is a limitation, although this hopefully doesn't affect too many languages.

It was also interesting to see that they store it in their own format, apparently for faster loading. Maybe OpenOffice.org and Firefox can also benefit from something like this?

Localisation on paper

2009-05-20T16:33:46+00:00

If I have to explain the term "localisation" to people, I always try to mention aspects falling outside of translation. Things like date formats and currencies are easy examples to use. Although spell checkers are commonly known, many people see that rather as part of language technology, although it is of course part of adapting a computer system to its users. There are, however, more aspects, with a few nice examples in the world of Free Software.

If you install a Linux distribution in Chinese, the system will install the input methods that make it possible to type Chinese. It involves software as well as "dictionaries" that is required for the input methods. Specific fonts will also be installed that might not be installed otherwise, especially because of the size (good Chinese fonts can take up quite a bit of hard drive space). Firefox gives the opportunity to specify special search engines for a language. GCompris has sound files that can be created separately for each language (who's going to help me with the Afrikaans ones?).

Another aspect that one is usually entirely unaware of, is paper sizes. With this I'm not referring to A4 versus A3, but the whole A system (part of ISO 216) versus the American system of paper sizes ("Letter", "Legal", etc.). I guess most South Africans have not yet even seen a sheet of "Letter" size. I don't even know if you'll be able to buy one anywhere. Anyway - we use A4.

These formats have different sizes, and computer programs must know this to be able to do page layout correctly. Mostly this is of no consequence to anybody. Dwayne created locale files for Linux and OpenOffice.org years ago for all the South African languages. These specify that we use A4 in South Africa. On Windows it works differently - it is configured per printer. Of course you can always go and configure these things again, just like with any other setting of the printer.

With this in mind it was a surprise when we realised that Firefox doesn't obey the locale information on Linux, and always uses "Letter" as the page size, even for the Afrikaans edition. In Firefox at about:config you can set the value of "print.postscript.paper_size" to "A4". I wanted to make this change for the Afrikaans version by making a change to the file firefox-l10n.js by fixing this value in JavaScript with:
pref("print.postscript.paper_size", "A4");
However, I was recommended not to do this based on some technical and administrative grounds. Of course it wouldn't be the correct solution (the software should get the setting from the locale data), but I really wanted to correct it for the Afrikaans users of Firefox. Hopefully this can get some attention at some stage. Possibly relevant bug reports at Mozilla:

Thank you to Fabrice Facorat for linking to the extra bug reports.

It is quite sad to see this if you take into account that the American system is basically only used in North America.

Conquering the CellRendererWidget

2009-05-11T09:11:42+00:00

It seems that there have been some attempts at creating a custom gtk.CellRenderer that can render an arbitrary widget. Wait or try no more, it is here!

I have recently spent a few days searching, learning and sifting through documentation and code in order to achieve this seemingly impossibility. It turns out that the best example of the cell renderer in question was Virtaal itself! The initial commit contained an example clean enough to reproduce most of the functionality. However, it still had the drawback of relying on a custom gtk.TreeModel class as well as a custom gtk.TreeView. The aim of this widget was to be generally usable as a stand-alone cell renderer.

Without any further delay, I present to you:

the code

(queue victorious orchestra music)

It is, of course, not flawless. Desperation, exhaustion and the looming weekend led me to use a hack or two to get it working in the end. Not ideal, but acceptable for the use(s) I have in mind, at least. Its size calculation is also not correct yet. This means that your widget(s) will get packed into a cell with the height of a cell containing text.

Flawless or not, I'm still very relieved to have gotten this far and I hope that this cell renderer will be useful to others. Please use and abuse as you wish and share any improvements (walter_translate_org_za - fill in the blanks).

Minority languages cut short in the SMS

2009-05-05T09:07:53+00:00

I read today on Slashdot about the reason for the length limitation of 160 characters in SMSs. It is an interesting story, and relevant to us today, since it is one of the reasons for the current outlook of the cell phone landscape. Our languages of course didn't escape unharmed.

We recently started working on a training manual for localising in Northern Sotho together with Mosekola. In places we tried to give attention to issues that are specifically applicable to cell phones, such as limitations on length. Nokia has had cell phones available in a few South African languages for some time. It is usually Zulu, Xhosa, Afrikaans and (Southern) Sotho. The first three are the three biggest languages of South Africa. Sotho is number 7 however. The main reason for this choice is probably that Northern Sotho uses the characters "š" en "Š" which aren't available in the simple set of charakters for SMSs. Interestingly enough even a few characters for Afrikaans are missing from this set, e.g. "ë". I know some cell phones can handle these characters, but I'm pretty sure that "ýïû" aren't handled anywhere, but would like to hear if I'm wrong.

Other plans have been made for many languages, especially in the Far East. For some reason my current phone can type Greek characters. Brilliant "strategy" from Samsung for the South African market.

It is sad that many languages simply doesn't have the market strength and public voice to request better technology. Without choice, normal market forces won't sort this out.

Monolingual translation formats considered harmful

2009-03-27T14:28:00+00:00

For a long time I've wanted to write something about monolingual translation formats. I've added a page about monolingual files in our localisation guide. The title of this blog post is of course more of a tribute to Dijkstra than the fact that I really consider them harmful, but they definitely have their issues. I would really seriously encourage all software projects to consider bilingual formats, or to ensure that they solve the mentioned issues with monolingual files if they stick to them. Any other issues with these files that we need to mention on that page?

Video of Virtaal's functionality

2009-03-03T16:00:12+00:00

I made a video that shows of some of the features of Virtaal - our translation tool. Download it here (33MB, Ogg Theora format). I hope it helps to make things a bit more lively than the mere text descriptions of the features.

Video van Virtaal se funksionaliteit

2009-03-03T15:57:54+00:00

Ek het 'n video gemaak wat bietjie van Virtaal se funksionaliteit wys. Laai dit gerus hier af (33MB, Ogg Theora-formaat). Ek hoop dit maak dinge bietjie meer lewendig as die blote teksbeskrywings van die funksies.

Firefox-style button with a pop-up menu

2009-03-02T09:56:01+00:00

(Clearly I've been neglecting my blog for the last couple of months. To try and change that, I'll start with this small post, just to get the ball rolling again.)

If it is unclear from the title, the Firefox widget I'm talking about is the toolbar buttons you get in the bookmarks toolbar that pops up a menu when you click it. I just wanted to take its functionality a bit further for Virtaal: Creating a similar widget which is not limited to a toolbar.

Google Translate + libtranslate + Virtaal = machine translation in 7 new languages

2009-03-01T19:18:33+00:00

Google has just added 7 new languages to their machine translation service, taking the total to 41.

I've submitted a patch to libtranslate which will make it work with these languages.

Of course Virtaal, Translate.org.za's localisation editor uses libtranslate, so now with the libtranslate TM plugin you can get machine translations for Turkish, Thai, Hungarian, Estonian, Albanian, Maltese, and Galician. This work on Virtaal work is being supported by funding from Mozilla Corporation and from IDRC through the African Network for Localisation, Anloc project.

Fixes for Skype Video, Webcam on Fedora

2009-02-04T10:11:35+00:00

I'm sure everyone has written about this... Might as well help to ensure that people can quickly find a solution.

env LD_PRELOAD=/usr/lib/libv4l/v4l1compat.so skype

Place that in your launch icon for Skype. Or drop the 'env' and run it from the command line. You should drop into Skype's option dialog->Video and test. Hopefully you now have shiny video in Skype

libtranslate, TM plugins and Virtaal

2009-02-03T22:25:41+00:00

Someone asked on the translate-devel list about integrating something like libtranslate into Virtaal. What a great idea! Except, its already done and then I realised I hadn't told anyone about it outside of the devel lists. So let me tell you what I did.

Virtaal 0.3, due in a few days maybe hours, added - thanks Walter - a plugin framework that allows various features to be coded as plugins, iIncluding of course translation memory or TM. Walter and Alaa created some that can access open-tran.eu and our own TM server. I was curious and wondered how hard it would be to add others, so I added two:

TinyTM - a TM solutions that we stumbled upon. Although they talk about linking with OmegaT and others on careful reading it turns out we're probably the first CAT tool client to be able to connect with TinyTM.
libtranslate - this uses the libtranslate library to query various online machine translation tools to get potential translations.

It took me a weekend to do this. Most of the time was consumed finding and fixing libtranslate compile issues to allow it to be used by Python's ctypes module. I spent very little time doing the actual implementation. With my TinyTM implementation I spent most of the time working out which Python module I should use for PostgreSQL access.

libtranslate: Many people have got and continue to get quite excited about this. Me, I'm not overly excited. I think it will prove useful but only as useful as the actual MT that are provided. How good are the MT, my guess is not that good. For a new translator we run the risk that they accept the suggestion blindly. For an experienced translator we may find that they can type faster then we can get the MT result. What I think is useful is that it can give someone a push in the right direction, so if you really don't know how to translate the segment you might get enough information to type your own translation. Time will tell.

TinyTM: nice name. They use PostgreSQL's built in Levenshtein distance module with some nice recursion ideas to speed things up. Potentially they can add other text matching idea. Personally I think the TM server we've built for Virtaal will give better results in the long run and be less database dependent, although we already use FTS on sqlite.

open-tran.eu: I didn't work on this module but I think its worth talking a little about what makes hte Virtaal plugin different from everyone else's. Some background for those who don't know about this resource. Open-tran.eu is a repository of all translations in the FOSS world (maybe not all but everything you'd care about). They use the Translate Toolkit for some manipulation into PO files, so they must be nice guys! You can query from the web and they also expose an XMLRPC interface for querying the database. Think of it as having an absolutely massive TM of all FOSS in all possible languages at your fingertips, neat. So we had to have a plugin to this resource. Except we didn't quite like the results. Open-tran does word based lookups, anything else would probably be too expensive on their server, this results in some matches that are just way off and not useful. Our solution was to mix the open-tran results with a bit of Levenshtein distance. In laymans tems it means we get quick results from open-tran for highly probably matches, we then reduce that list through Levenshtein distance matches and take those results which have the best final matches to you the user of Virtaal. We let open-tran serve us without extra overhead and rather let the client software do the hard crunching. My feeling is that its open-tran done right.

Localisation Information Language - preventing mistakes and increasing the richness of localisation

2009-02-02T13:13:31+00:00

You might want to call this an announcement of sorts. This is something that has been brewing in my mind for many years now, its just that I got irritated, found the time, sat down and began writing and planning.

I'm trying to design an language or system that allows localisation information to be conveyed to a translator. A language that is simple enough for a human to understand and act upon such as "dont translate word 'IMAP'". Yet regular and well formed so that it is machine readable so that a localisation tool can understand the commands and take action on them. In this case it can test that the acronym IMAP has indeed not been translated.

The initial thoughts for this where inspired by the LOCALIZATION NOTES in Mozilla source code. In the Translate Toolkit we use the DONT_TRANSLATE command to drop messages that should not be translated. So we reduce the burden for a translator in that they never see those messages, ever. Saving them time. But in my mind these notes have never reached their full potential and this project is an attempt to push localisation notes further. This kind of inteligence forms part of our funded work for Mozilla and I'm keen to see if I can't use this towards my studies.

I've spent quite a bit of time fidling with grammar and layout to get something I feel is usable. More about that in later posts.

Sorry that I'm a bit sketchy on the commands and grammar, they are still in a state where I move large chunks into common commands as I discover commonalities. For now let me describe in text mostly of what I've got covered:

Whether you should translate something or not
XML localisation: which parts can you change
Character set limitations: what are the characters that you may use
Lengths of strings: how long or short a string may be
Localised items: numbers and other items that should be localised but you wish someone had told you
Variables: a system to allow examples and descriptions of variables and to identify non obvious variables
Configuration options: sizes, valid options, etc

That's it for now. I'm still investigating things like: credit entries, language configuration entries and others.

I have found this quite a fun exercise. I'm convinced that a language like this is more powerful in the hands of translators then programmers. The simple reason being that programmers simply don't understand all the issues localisers face. While translators would love a way to leave instructions for those coming after them, these could be people on their own team or in another team, that prevent them making the same mistakes.

A simple example, of something I found in OpenOffice.org. OpenOffice.org has 11 styles of variables, pofilter can test for all of them. Yet in one message the word DOMAIN apeared in capitals. It wasn't an OpenOffice.org variable yet it just looked like a variable but you could not be sure. Checking in other translations it was a 50/50 split as to who had translated it and who hadn't. Finally after getting some feedback from the developers we learnt that yes it is a variable and thus should not be translated. The question is how many people have localised it? How many had to invetigate the issue to discover over and over again that they should not translate the word DOMAIN? And how many new localiser will continue to translate the word because there is no comment to remind them of the finding. This localisation language would allow the new localiser to not make that mistake and would allow tools to check that no further mistakes are made.

Fortunately with FOSS a translators can get such a comment included in the source text and in such a way communicate valuable information to the 50 other language teams that follow them. Thus since translators are the people who care, and FOSS allows them to care I'm convinced that localisers will be the best implementors of such a system.

I've worked hard to think about how we ensure that the comments can be placed at the root. Thus they must appear in the DTD files, in the C code and they end up in PO and other intermediate formats after their extraction. Thus they have some permanance in that they ride with the code.

With this language we should be able to do a few things that have not been possible until now:

Eliminate all errors in configuration settings: Sounds impossible but by preventing errors as they're made and easily and quickly checking for them I'm convince we can eliminate all of them. If some remain we need to extend our localisation language.
Produce more complete localisations: as we can communicate to translators that they may and should localise things that in the past they have been too scared to do such as a date or a number we end up with localisations that are more closely aligned with the cultural conventions of the language.
Produce higher quility translation involving variable data: by communicating more clearly which parts of a message are variables, what they do and examples of them in use we can allow a translator to fully grasp what they will be producing.
Reduce or eliminate the complexity and limitations of some formats: work around localisation issues in certain source formats such as accesskey in Mozilla and sentences broken across multiple strings.

The language tries best to be backward compatibile, thus a DONT_TRANSLATE note from Mozilla source although deprecated in the new system will still work.

Large amounts of time were spent reading the XLIFF spec. To be honest getting a bit depressed about it at the same time. But it was a good exercise to get an understanding of the types of localisation information XLIFF is able to store and convey.

My general plan moving forward is to do the following:

Continue to flesh out the syntax of the system, eliminate duplication, increase human readability and consistency
Review all Mozilla files, OpenOffice.org and others to find common issues that are not covered and that can be handled by a machine
Build a parser in Python
Add some tests to pofilter to see if my concept works as expected
Longer term: I'd like to take these findings to XLIFF and improve the QA component of XLIFF. With the vision for XLIFF 2.0 being to create a skeleton format on which to hang various extension I would see this language and QA in general as an ideal extension for XLIFF.

I hope that in the coming few weeks I can get a draft out for other people's input. For now please leave your comments, I'd especially like to hear about any problems that you encounter that would be worth trapping. Just a heads up, no this is not an l20n like project, I'm not trying to solve the issues of language, declention, gender, etc. Others who feel the pain of those issues are working on them and the two are certainly not mutually exclusive. For me localisation information interchange has some really low hanging fruit that all can benefit from now and we have the tools that can implement the checks that are needed so I'm pushing ahead in that direction.

So forward to healthier localisations environment with - Localisation Information Interchange - shall we call it LILy?

Don't forget to leave comments about your current pains and lets see if we can get them integrated into LILy.

Language and dialect codes

2009-01-12T09:53:33+00:00

We recently had an interesting conversation in the office about language and dialect codes. If we store translations in translation memory, how should we handle 'en' (English) versus 'en-US' (English in the USA)? Should we assume that it is the same, or should we handle it as two entirely different languages? The problem is not quite as simple as one would have hoped for.

xclip - where have you been all of my life!

2009-01-06T09:15:02+00:00

I found xclip. It wasn't like she'd gone missing and I can't for the life of me work out why I never looked for her.

xclip allows you to take command line output (say an error log) and push it to the X clipboard allowing you to simply past the data onto a webpage (a bug report for instance). I'm rather embarassed to admit that I used to open a file in vim and copy and paste blocks of data, I feel like my repressed Windows gene just made me blind to a better way.

Virtaal on Fedora

2009-01-04T22:40:00+00:00

Virtaal is now packaged for Fedora. The real thing i.e. it will appear in the Fedora repository. The packages are waiting to be pushed to stable for Fedora 9 and 10. Please download and test these builds (F-9 or F-10).

I'm hoping to build versions for EL4 and EL5 but I'm in no rush for those.

Re: Bringing all translation management tools together

2008-12-15T14:29:49+00:00

Djihed Afifi recently wrote a nice web blog entry about his dream of bringing the current FOSS translation management tools together. Instead of replying there, I thought his probably deserves something a bit more thorough reply.

A view on the use of local languages with Google Zeitgeist

2008-12-13T13:44:41+00:00

Today I had a look at Google's Zeitgeist (spirit of the times) for 2008 which gives an indication of which searches were popular in which countries. It was quite interesting to notice which countries have lots of searches in English even if English is not a big local language.

Translate Toolkit on Fedora. Status of Virtaal and Pootle

2008-12-12T08:08:55+00:00

Well not really :)

The Translate Toolkit has been packaged for Fedora for a while now, what I've done is update to v1.2 (with v1.2.1 in the pipeline) for both Fedora 10, 9, RHEL 5 and OLPC. This is the groundwork needed to get first Virtaal and then Pootle into Fedora.

Virtaal's MVCisation

2008-12-08T09:53:51+00:00

Over the last two weeks Virtaal's code has been refactored to be more easily extendible. Changing Virtaal to a MVC architecture proved to be quite an intense and interesting endeavor (my GTalk status was "Deep hack mode" for most of these two weeks), but well worth it. And here we are! Virtaal's MVC branch is just about on par with the trunk version.

Pootle eats Django

2008-11-22T08:06:57+00:00

Pootle and Mozootle have been merged. Now its time to migrate Pootle onto the Django web framework.

No more Microsoft Internet Explorer for the 2010 soccer world cup

2008-11-22T07:42:31+00:00

The only web browsers gaining more market share by 2010 is everything except Internet Explorer, with Mozilla Firefox leading the pack. If you are planning to be at the soccer world cup in 2010, planning to deliver your web content to a 2010 audience, then these trends are very important. They influence what technology decisions you need to make today to deliver to your 2010 audience.

So lets project current browser usage trends over the last year up until the start of the world cup in South Africa, June 2010, to see how that future audience will browse the Internet.

This is what I found (my data source data from Net Applications):

Internet Explorer - still the largest market share holder but now commands only 61% of the market (it drops to below 60% in the following quarter). A far cry from those days of 95%.
Mozilla Firefox - commands over 25% of the browser market. A full quarter of your potential audience.
Alternate browsers (Firefox, Opera, Safari) - 1 in 3 people will use these browsers.

How does this influence your technology decisions:

Must work on all browsers - If you don't you get to lose 1/3 of your audience. Relatively easy if you ensure that you follow W3C standards and use AJAX abstraction libraries.
Forget anything that only works on one browser - Really this means forget anything that only works on Internet Explorer. If you decide to target only Internet Explorer and succesfully draw 6 million customers remember that it could have been 10 million. Your business and marketing people will kill you, and rightly so. So no Microsoft Silverlight for you.
Forget about new technologies. I'd say if you are doing video go with flash, its proven. If you do want to innovate use native video (It will be in Firefox 3.1 and is part of HTML5, other browsers will follow soon). There is really enough space for you to create an innovative experience with AJAX and existing technologies that work now.

Caveats, there are always caveats. I've based this data on the current average monthly changes in market share. Many things can happen to change these both up and down. Events such as:

Microsoft might start innovating (or marketing) in this space - this will be neutral or slow the drop, its unlikely to result in clawing back of market share. People who use Firefox, Chrome or Opera make a concious decision to use the product. They would thus need to undecide, which is seems highly unlikely.
Many more people move to cellphone usage - our figures are based on desktop browser products. But certainly we don't expect any positive shift for Internet Explorer, since the browser engine that dominates on cellphones is Opera's (and maybe Safari with all the iPhone users).
Google Chrome gets used by more the 1% of the market - the current data is so low that I wasn't able to project this in any useful way.
We'll see new versions of browsers over this time - The market share figues may stabalise or the trends shift slightly. This will leave us in 70/30 split between Internet Explorer and the alternatives. I don't think that changes your decisions much, its still means you could have had 10 million users when you landed 7 million.

See you at the world cup!

Things that changed the way I code

2008-11-20T09:42:17+00:00

I believe that all code monkeys (should) strive to be the best that they can be. Part of doing so is adapting your own process to maximize productivity and to make things easier for yourself. This is just a spur-of-the-moment post about some of the specific things I have discovered/changed in my few months at Translate.org.za that has had a big influence on the way that I develop software.

FIrefox locale language landing pages

2008-11-11T06:45:54+00:00

Mozilla has announced localised Firefox Download pages. Try the tasty Afrikaans page.

All I can say is that I'm very happy today, we've been frustrated by this for a long long time. But the new l10n teqam is listening. I'm greatful for that.

Now to the task of convincing Axel that en-ZA is localisation and would allow us to corner another section of the browser market in South Africa.

Blurred vision at Beeld

2008-11-09T17:35:33+00:00

I have repeatedly been disappointed by the low standards at Beeld (an Afrikaans daily with wide distribution including the area I live in). Apart from language issues that surprise at times, they fall far short of my expectations on the technical side as well. I recently encountered a problem with their character encoding — the reason I'm writing now.

Virtaal on Fedora: not just yet, but soon'ish

2008-10-30T10:22:37+00:00

I've just completed getting python-Levenshtein packaged for Fedora, it's currently being pushed to Fedora 9 you can help by testing and commenting on the package.

Its easy(er) with Kulula

2008-10-24T13:28:03+00:00

Earlier I had something to say about my experience with Mango, and promised to write a bit about an alternative airline in South Africa. Apart from the fact that Kulula actually worked (in contrast with Mango), they surprised me with a touch of localisation. I dream of a day where such a half-baked attempt will look bad, rather than good.

When I tried to book my ticket at Kulula.com, I was (mostly) pleasantly surprised to see that the interface was slightly localised into Afrikaans.

Fennec in Afrikaans

2008-10-21T12:00:13+00:00

With the release of a desktop version of Fennec, Mozilla's new mobile browser, it is now possible for me to really test and experiment with it. I'll leave the general browser discussion to Tectonic.

Virtaal on the Mac

2008-10-15T07:43:40+00:00

One of the aims of Virtaal is to exploit the cross-platform nature of Python and deliver a platform neutral translation tool. This till means that we want to tool to feel like it belongs on the platform, which requires a little bit more work.
The Translate.org.za team does all of its development on Linux so we are porting to the other platforms. Let me give you some ideas of what we've done and issues that we've seen.

Autocompletion during translation

2008-10-10T14:39:25+00:00

Afrikaans (and many other languages) sometimes have long words. Some long words occur frequently within a single translation. This is a situation where Virtaal helps quite a bit.

Virtaal - a translation editor for the sake of language

2008-10-03T09:48:01+00:00

We released the first release candidate of Virtaal yesterday to the public. See the announcement on the mailing list. "Virtaal" is an Afrikaans play on words meaning "for language", but also refers to translation. I take responsibility for coming up with the name.

Switching from Subversion to git

2008-10-03T09:37:57+00:00

Working with Mozilla's Mercurial l10n repositories, I have gently been forced into learning about this type of distributed VCSs, and what a revelation it turned out to be!

Having only used Subversion (and a negligible bit of CVS) before, the ideas and principles of a decentralized VCS seemed chaotic and too cumbersome, to say the least. After reading (most of) the Mercurial Tutorial, however, my fear and reluctance turned to intrigue.

Why a DVCS?
For me it was the ability to do local commits that really caught my interest. Although local commits have been linked to so-called "cave coding", I still believe that it has more pro's than cons.

For someone reasonably serious about the quality of the code he commits, it is very irritating to have to commit changes that I'm not yet comfortable with. For example, I start refactoring some key component while I'm not sure if I will be able to complete it in time for the next release (or some arbitrary deadline). In stead of having to commit such serious changes to the public repo, I created a new branch and kept the changes there. When I finish the refactoring I can always push those to the public repo without having to worry much about what happened since I started the refactoring.

Because you keep a local copy of the entire history of the repository, any and all actions are extremely fast. No waiting for svn logs or slow commits, especially committing to Sourceforge from South Africa. With a DVCS you actually have control over your communication with the public repository server.

Of course, there are many more reasons, but I'll leave that for another rant. :)

Why git?
Playing around a bit with Mercurial, I eventually convinced myself to take the plunge and start using a DVCS. I still don't really know the difference(s) between Bazaar, Mercurial and git. My decision to go with git was based on the following advantages it had over the others, even before I began:

I have a quick, local git reference guide: Wynand! He actually helped me understand a lot of the core concepts of a DVCS when I was working with Mercurial.
Subversion integration. Our projects are hosted on Sourceforge's Subversion servers, so this is kind of a crucial point. git-svn came in and saved the day.
Awesome tools. This is related to the first point seeing as Wynand is the one who showed me the git tools' awesomeness.
Reputation. If it's good enough to be used for the Linux kernel, it must be good enough for a lot of other things.

Living with git
I've been using git for about 3-4 weeks now and it has been amazing! I admittedly only know about 10% of what a beginner git-user should know to use git effectively and already it has changed the way I develop software. Here are a few things that really stood out for me:

git branches are cheap. As described above, branching in code in your local repo is not only super-useful, but also super-easy and extremely powerful. You can do just about anything across branches on the commit level: merging entire branches, cherry-picking specific commits to apply to another branch. The list goes on... and on and on.
The git-gui tools. The two I have used most so far are git-gui blame and git-gui citool, the latter of which is something that has probably saved me hours already: Among its features, but most important to me, is the ability to easily select individual "hunks" (sections of a patch) that should be included (staged) in the next commit. Having a commit policy requiring as much atomicity as possible, this is a life-saver!

Conclusion
git allows you to do your development according to your own zen, whatever that might mean. If I had to summarize git in a few words it would be "powerful", "fast" and "control". Other DVCSs may very well have similar and/or better features, but I am, at least for now, happily ignorant thereof and content with using and abusing git. I would definitely recommend it to any programmer.

ODF-XLIFF starts working!

2008-10-02T09:17:44+00:00

We have just completed two new tools which will make the translation of ODF files a breeze. The tools are being developed in ~~a branch~~ the trunk of the Translate Toolkit (located at ~~https://translate.svn.sourceforge.net/svnroot/translate/src/branches/translate/odf-xliff-first-try~~ https://translate.svn.sourceforge.net/svnroot/translate/src/trunk).

Almost there - Virtaal and OpenDocument

2008-09-30T16:11:05+00:00

Friedel, Walter and Wynand have been busy releasing new versions of the Translate Toolkit, Pootle and spoiler warning a new translation tool.

Also Wynand has been working on a OpenDocument Format converter. I've just tested it on the OpenDocument 1.2 draft7 spec. Nothing like taking a huge document and putting the converter through its paces. Its early days lots of custom [[placeable_N]] in there. Check out Wyand's branch called odf-xliff-first-try.

Here is a picture of the two together:

Spelling Rulz

2008-09-26T15:36:52+00:00

I attending the PanSALB terminology workshop over the last two days. Bad timing with all the work I have to do. I'll hopefully write more about the event and my impressions later.
The good news is that PanSALB and the various National Language Bodies (NLBs) launched their revised spelling and orthography rules for the 9 (who know how we should group them) languages i.e. not English (we don't care for spelling) and Afrikaans (the AWS has been regularly updated since the 1920's).
Well done to all those involved.

The birth of the GNU generation

2008-09-08T06:06:29+00:00

GNU is 25 years old and the FSF celebrates with a video from Stephen Fry (which you can translate). And Mako reminds us that a generation is 25-30 years thus we have a whole generation that has been birthed in the life of this software. Perhaps my daughter's generation will wonder why people didn't see Free Software as obvious.

Vrot Mango

2008-09-04T14:45:24+00:00

("Vrot" is a South African word meaning "rotten")

Since I earned a nice holiday, I had the doubtful privilege of having to buy an airline ticket to Cape Town. That is not something that one would normally write about on a weblog, but my experience with Mango unexpectedly became relevant to the topics of this blog. Read more to see the story unfold.