Simplify translation process with Weblate

rob006

I wrote a simple system to simplify translation process by introducing a graphical interface and automation of some tasks. It was created mostly for Polish language package, but it should also scale for other languages too.

It stores configuration and all translations in separate repository and works in this way:

It reads sources for translations from configured plugins.
It saves source strings in sources directory and generates translation files for all configured languages.
Updated repository is fetched by Weblate (this is my private instance of this system, but it should be able to host Flarum translation). Weblate is an open source system which provides a graphical interface for translations and many cool features - it should make translations much simpler and more efficient.
Every translation done in Weblate should be committed to main repository with translations. So all translations data is always in https://github.com/rob006-software/flarum-translations.
Translations could be exported to separate repository - I'm already using this to generate translations for https://github.com/rob006-software/flarum-lang-polish. The English language is used as a fallback to bypass this bug.

This may look complicated, but everything except adding new extension/language and actual translation (using Weblate) could be automated. From extension maintainer perspective the only thing that he needs to do is to add his plugin to configuration - the plugin strings will be automatically fetched to all language packs and at worst case extension strings will not be translated (English strings will be used as a fallback). Language pack maintainer will no longer need to track all extensions - all new string will appear automatically in Weblate, and (with some config) all translations will be exported to his language pack extension.

The question is: are there any language packs maintainers who wants to work in this way? Any feedback about this approach?

rob006

I managed to automate most of tasks and now Polish language pack in managed by this system. All translations made at https://weblate.rob006.net/ are automatically exported to https://github.com/rob006-software/flarum-lang-polish/ (including README updates).

Some cool features:

Zend mode which makes translating much easier (including finding invalid translations):
Full view shows some useful info, for example how the same phrase was translated in other places:
Weblate is able to detect some basic inconsistencies between source and translation:
Badges with translation status in readme:

If you're language pack maintainer and you want to move your translations to Weblate, please let me know. There is a simple tool for exporting existing translations to repo used by Weblate, so there is no need to copy&paste all translations using Weblate GUI.

rob006

UPDATE

I partially automated the process of adding new extensions to the collective. Right now there is a script which searches for new packages at Packagist with flarum-extension type. A PR is opened for every new extension - it can be easily accepted or rejected. Using this script I added almost all compatible (and with translations) extensions to the repository - there is over 100 of them.

Meanwhile syncing translations strings for already supported extensions seems to work fine - new strings from upcoming beta 10 are already available to translation.

rob006

I updated Weblate instance to version 3.9 (from 3.7.1). It brings improvements and new features but above all it improves performance of some views, so interface should be a lot more responsive.

Also, some time ago I created basic FAQ about this project and fixed description in README.

rob006

A couple of new features and improvements have been added in last week:

Script for fetching new strings from extensions was updated to prefer stable releases over branches. It turns out that it is pretty common for extensions to make some backward-incompatible changes in master branch and never release it. Since now there should be no situations where translations were adjusted for an unreleased version of extension, which makes them incompatible with last stable release. New strings still will be fetched from default branch as long they're not conflicting with stable release.
I created command to generate releases. It generates changelog entry, publishes release (following semver) on GitHub and generates an announcement for forum discussion. You can see example for Polish language pack: template, changelog, release, and discussion on the forum.
I added pages with translation status - list of extensions ordered by a number of monthly downloads (see example for Polish). It should help to decide which extensions are the most popular and should be translated first.
Subsplit command no longer adds English fallback for missing translations - since flarum/core1961 was merged, it should be no longer needed in next version of Flarum.

rob006

rob006 I added pages with translation status - list of extensions ordered by a number of monthly downloads (see example for Polish). It should help to decide which extensions are the most popular and should be translated first.

Since now these pages display trends - script compares current download stats with stats from last week and add badges with differences. Since badges are images (GitHub limitation) it could take w while to load the whole page.

Littlegolden

This is great! I'm going to use it after these busy days.

rob006

Littlegolden Open an issue in https://github.com/rob006-software/flarum-translations/issues when you'll be ready, I prepare configuration for your language. I can import translations from existing language pack, but I need to know which repository use as a source.

rob006

Thanks to luceos, premium extensions are now available to translate. 🎉

rob006

Premium extensions are now automatically detected by bot, and they're included in translation status page - you can easily decide which premium extensions are the most popular and should be translated first.

It was possible thanks to @luceos who provided access to Extiverse API. Thanks! 👍️

luceos

rob006 you're welcome. It's the least I can do for such an awesome project in the ecosystem like yours!

rob006

Minor change - all translations are now handled by single Weblate project (previously there were separate projects for core, FoF and community extensions). It should make navigation and searching easier.

tolgaaaltas

Do you think to update your Weblate? v4 looks really good.

rob006

tolgaaaltas Do you think to update your Weblate? v4 looks really good.

I will definitely update at some point, but right now this is far more complicated than just upgrading Weblate. I have a lot of stuff on this server and it is kind of like a chain reaction where upgrading Weblate will require upgrading server version, which will require upgrading other apps installed on this server. Still I hope I will be able to do this in first half of 2021.

SKevo Could this work as a webpage? My idea was to create a web interface with a bunch of translation files and languages where everyone can contribute (something like Crowdin)

What do you mean? Weblate already works like a web interface where people can contribute.

SKevo

Really cool, why didn't this popup in my front page earlier?

A while ago, I had the same idea - make Flarum translations easier using some interface... But I couldn't do it alone, and yours looks great!

Such tool definitely needs more attention! Could this work as a webpage? My idea was to create a web interface with a bunch of translation files and languages where everyone can contribute (something like Crowdin)

SKevo

rob006 never used it so I didn't know, sorry. I just read your discussion 😛

rob006

Since today new feature is available for translation status summary (example for Polish) - I've added compatibility indicator. It is based on constraint for flarum/core in composer.json and recognizes 3 states:

🟢 - compatible: it is compatible with last 2 releases of Flarum and incompatible with release before last "big BC break" (like beta 13). This is usually extension under active support.
🟡 - unknown: it is compatible with last 2 releases of Flarum, but also compatible with release before last "big BC break" (like beta 13). While extension may work in such case, it is usual abandoned extension with too wide constraint for flarum/core (like ^0.1.0-beta.8), so while you can install it with recent Flarum, it may not work.
🔴 - incompatible: This extension can't be even installed with recent Flarum due to upper bound in constraint for flarum/core (like>=0.1.0-beta.10 <0.1.0-beta.14).

Now it should be easier to decide what extension may not be worth to translate.

In next 1-2 weeks I plan to remove most of incompatible extensions (🔴) and some of probably incompatible (🟡) with low number of installations. I will share list of affected extension few days before actual removal.

rob006

rob006 In next 1-2 weeks I plan to remove most of incompatible extensions (🔴) and some of probably incompatible (🟡) with low number of installations. I will share list of affected extension few days before actual removal.

rob006-software/flarum-translations465

askvortsov

rob006 By any chance, do you know if Weblate could be a candidate for translating large pages of text, like with the documentation (https://github.com/flarum/docs)?

rob006

askvortsov You would need to split document to smaller translatable parts and put into format recognized by Weblate. There are tools for that (like po4a), but I'm not sure how reliable they are - you would need to test it on actual documentation to spot any problems. So while it is possible, Weblate itself does not support this usecase and there may be better tools for translating documentation.