Common L10n stumbling blocks

Dominion

First, let me say a few words about the purpose of this thread.

Like my profile blurb says, I'm a translator, not a programmer. Most of my localization experience has been with projects that were finished, or nearly so, by the time I got involved. And frequently I've had to ask the devs to review a bit of code that they thought they were already done with, because it was outputting info in a way that just doesn't work in Japanese (or English). The subsequent haggling over how to handle such issues generally means extra work on both sides, and if I have occasionally found this process frustrating, I can only imagine it must be more so for the devs, who have to deal with many translators working in many different languages...

Having recently decided to localize esoTalk just as Flarum was being developed, it occurred to me that I might be able to help out with Flarum even before the actual localization process begins. The idea is to help @Toby and @Franz work out some general i18n guidelines that could allow them to avoid l10n issues before they happen. I mentioned this to Toby, and he asked me to post here. And so, having finally got my esoTalk site up and running, I'm ready to do that.

The problem is: How to begin? I haven't had a chance to look closely at Flarum yet, and since I've got a book translation project hanging over my head, I doubt I'll get that chance very soon. This makes it hard to address specific issues in Flarum (which is still a moving target in any case). What I can do at this point is talk about the sorts of things that have caused issues in past projects, and suggest some approaches that can help avoid such issues. Hopefully that will be of some use.

Since I expect to be translating Flarum into Japanese, the examples I cite will involve that language. Hopefully other translators will chime in with examples from other languages. In any case, please understand that the aim here is not to argue for special provisions that need to be made for any particular language, but rather for special provisions that need to be made so that Flarum can be translated into as many languages as possible. I'll just be talking about Japanese because it's what I know best.

In the same vein, I think it will be easiest to illustrate my points by referring to issues I encountered while working on translations of esoTalk and FluxBB (as well as other similar applications). My intention is not to "name and shame" those applications or their devs, or to suggest that they should've done better. Natural language is so strange and varied, it's probably impossible to provide for every strange thing it can do. The goal here is simply to offer some feedback from past l10n experiences in hopes that it will help make the Flarum l10n experience a bit smoother.

Finally, I've been chewing on this topic for a while, and have a fair bit to say about it. More than will fit in one post, really. So unless someone else jumps in and adds to the discussion, I could end up responding to myself a few times in a row. I beg your indulgence and your patience!

Dominion

Goals of the i18n process

Before getting down to individual cases, I thought it might be useful to talk a bit about what we're trying to achieve here. When evaluating any potential i18n guideline, it may be helpful to think about how much that guideline helps to achieve one or more of the following goals. (This list isn't set in stone, so if you can think of any additions or changes, please say so.)

To define the limits of the l10n task

We want each language pack to be a complete l10n solution; that is, we don't want to have to include hacks or other extensions in order to make a language pack usable. This means that the devs have to either ensure that the language pack contains everything localizers need to do their jobs, or be prepared to work with localizers to solve issues on an ad hoc basis. The latter approach is problematic for a number of reasons, so ideally, the localizer's job should begin and end with the language pack.

To make the localizer's job easier

This ties into the previous goal, in that it's best not to assume that the localizer knows anything about programming. Sure, there may be folks out there who can explain why a certain line of PHP is causing issues, and perhaps even suggest an alternative (I'm just getting to that point), but some of the best localizers may be nothing more than translators. Anything that helps translators do their job without asking for changes to the code will ultimately increase the pool of translators who can work on the job.

Of course, this also applies to other, simpler things like avoiding unnecessary duplication of language strings. Anything that makes the job harder will make the project less appealing to translators. Granted, Flarum will have fans who want to localize it no matter how hard the job is. But there are also folks who will choose a forum script based (at least in part) on the amount of effort needed to localize the application, and take the path of least resistance. Making the job look more appealing will increase the number of translations available, and therefore the number of people who use the software!

To level the field for localizers

This is the Really Big One. It's easy to internationalize for languages that are relatively similar to English. (After all, we wouldn't need to worry about Unicode if everyone used the Latin alphabet!) If you want Flarum to be used globally, however, that's when things get tough. It would be ideal if we could make sure that no localizer would have a tougher job translating Flarum than any other. That's an awfully tall order, since (as I've said) natural language does some strange things and it's hard to be prepared for all of them. But it's the goal we ought to aim for.

To give localizers the freedom they need to localize their way

This one might be a bit controversial, since it may be necessary to restrict localizers in some ways to maintain a unified product image. But here I'm not talking so much about maintaining standards of usage and quality, as I am about making it possible for translators to set and meet their own quality goals.

For example, things like bits of hardcoded text can interfere somewhat with the finished look and feel of a translation, but not so much that it becomes unusable. The result may be fine for a hobby site, but less desirable for use on a corporation site, where a lack of polish could be considered detrimental to the corporate image. So an internationalization process that removes those obstacles, and gives localizers the freedom they need to achieve any desired level of quality, will increase the number of ways and places that Flarum can be used to good effect.

To allow for systematic handling of issues

This is where I get a bit fuzzy, because although I'm aware of many of the difficulties involved in localization, I'm not quite as aware of all the things devs do to deal with those difficulties. But I have gotten the sense that some changes made to the code to deal with language-specific issues can be (for want of a better word) a bit kludgy, and that the gradual accumulation of such changes can make the code harder to maintain down the road. So I'd guess the goal is to head such issues off at the pass, so to speak, with a more elegant solution put in place beforehand.

EDITED: Forgot to add the one about freedom.

Toby

@Dominion Woah, this is awesome. Thank you so much for your enthusiasm here. Really looking forward to working with you (and other translators) to provide really good l10n support. 🙂

I agree with all of the goals you've listed.

Dominion

Thanks Toby! ... Unfortunately it seems I missed the one about freedom, so I've gone back and added it in.

Incidentally, this is doubling as some fine Markdown practice for me. I'm beginning to like it more. Moving on...

Stumbling block 1: hardcoded text

I thought I'd start out with the most obvious one, and so get it out of the way. These days most forum scripts seem to do a pretty good job of keeping language strings out of the code. But it's still not unusual for little bits of text to creep in unnoticed.

The most common culprits are things like punctuation, such as commas and colons, and symbols such as parentheses, square brackets (which I've seen used around links to make them look more like buttons, something that is better done with CSS), and even the "@" symbol (which was memorable for reasons I'll get to in the next post). Oh, and spaces. You still run across the occasional hardcoded space now and then.

The great majority of these bits of hardcoded text won't cause troubles for many translators, because they're used the same way in many languages. When it comes to a language like Japanese, however, they can cause all sorts of problems. In some cases (such as the comma), the symbol just isn't used in that language, or it's replaced by another symbol that does the same thing. And in nearly all cases, Japanese has its own full-width glyphs that do the same things. Using the half-width versions we're more familiar with makes the page look a bit funky: compared to the Japanese text, the colon looks too small and too low, the parens and brackets are scrunched up too close to the characters they enclose, and so on. The issues are largely aesthetic, but in some cases the legibility of the page can go down by a significant amount.

Even hardcoded spaces are trouble, because Japanese doesn't use spaces to delimit words ... so why start doing so here, and here, and here? That soft of thing. Ultimately, this sort of thing can make the finished translation look a bit unprofessional. That would be especially unfortunate with Flarum, which is so slick-looking to start out with!

As a general rule of thumb, then: The first rule of i18n is, hardcode nothing!

... No, I'm not laying down rules. I just couldn't help adding the obligatory Fight Club reference. ;-)

Dominion

How is that a Fight Club reference, you ask? Well, you have to read a bit further.

Because, as you've probably guessed: The second rule of i18n is, hardcode nothing!

Yep, we've got some more talk about hardcoding in our future. But this is a different kind of hardcoding, one that's more subtle and easier to do without noticing that you're doing it. I'm talking about...

Stumbling block 2: hardcoded syntax

Sometimes code is written in a fashion that imposes a specific order on the words, which may or may not work in other languages. The change I suggested for how a list of user names was being imploded in esoTalk provides one example. (For readers other than @Toby it goes something like this: In English, a list of user names looks like: John, Paul, George, and Ringo. Japanese people, however, would rather see: John and Paul, George, Ringo. Since esoTalk was coded to always insert the and between the last pair of names, it was necessary to adjust the code to enable movement of the and to its proper Japanese position.)

As another brief example, I've seen one forum script (I can't recall which one) that used the "@" symbol to timestamp posts. That is, it would display the legend "username @ date/time" after the post title. Now, the hardcoded "@" is a bit problematic, because it's shorthand for an English word. (I won't say that most Japanese users wouldn't understand it, but it might cause a few puzzled frowns along the way, and ultimately it would look out of place in a translation where fluent Japanese is the goal.) But it's also a problem because the word order is off. In Japanese, time-related expressions usually come first, not last.

Let's super-size the problem by looking at the typical word order for a whole sentence. In English, we might say:

I posted a comment in the Flarum forum at noon yesterday.

If we tried to rearrange these English words to resemble a Japanese sentence, it might look something like this:

Yesterday's noon-at Flarum's forum-in comment posted.

As you can see, it's pretty much backward. So to make a Japanese translation possible, any code that can output a sequence of strings in an order approximating the English sentence (or any subset of its elements) also needs to be able to handle them in reverse order. What's more, the grammatical functions indicated by prepositions are indicated by postpositions in Japanese, so even on the phrase level, it needs to be possible to turn things around. And while Japanese doesn't have to worry about things like pluralization and gender agreement, we do have postpositions indicating grammatical case (subject and object), so we may need to provide for that. And since the subject of a Japanese sentence is frequently omitted, as seen above, the translator needs that option as well. And so on...

The upshot is: any output containing bits of language that participate in grammatical relationships to each other is best handled as a complete sentence (clause, phrase, whatever). When the code breaks that whole up into chunks and forces a specific syntax on the output, you can end up with a translation that's beautifully fluent in most places, but now and then, despite the translator's best efforts, ends up talking like Yoda for no reason. Comprehensible, the result may be; polished, far from, it is.

The use of the hardcoded "@" above to indicate a relationship between the poster and time of the post is an example of a case where the code forces an English syntax on the output. As another, I would cite the way esoTalk displays text in the "Sort By" button on the member list page. That button places the verb first, followed by one of a selection of nouns (name, posts, last active) indicating the sort criterion. Since the verb should come last in Japanese, I ended up having to translate "Sort By" as "Sort Order" followed by a colon. It's not a major problem, but polish-wise not on a par with the rest of the job. So in that sort of situation, as well, extracting the whole button name as a string with an %s where the noun goes would provide some useful flexibility.

One final thought, while I'm on this subject:

There were quite a few places where I found myself doing some linguistic gymnastics while translating esoTalk. (It never got quite to the point of Yoda-speak; more like using the passive voice when the active would have been more natural.) The main reason was that many strings used arguments in a specific order (e.g. "posted by %s at %s" and the like). Back then (oh so many weeks ago!) I didn't know that I could swap the placeholders and write "%2$s -at %1$s -by posted" if I needed to. (Here, of course, I'm assuming it's possible to do that with language strings; I could be wrong.)

Perhaps, as a general rule of thumb, it would be a good idea to use the order-specific notation by default, to point PHP-challenged translators in the right direction? Lacking that, perhaps it wouldn't be a bad idea to whip up a "Flarum Translator's Guide" with a summary of such basic techniques, to help translators over the learning curve. (It could also include a brief YAML guide and such.) Or at least a collection of links pointing to the necessary information.

Just an idea I had in passing ...

PS: Come to think of it, the Translator's Guide might be a good idea in any case. Catch me in the right mood, once I've had a crack at translating Flarum, and I'll probably volunteer to write one up. ;-)

Dominion

Intermission

In reading over what I've written so far, I get the feeling I've been focusing too much on how these stumbling blocks affect me as a translator, and not enough on drawing conclusions that may be useful to the development process. So please allow me to add a quick TL;DR before I move on to the next topic...

Hardcoded text:*

The conclusion to be drawn here is fairly simple. Anything that gets printed (other than what comes from the database or the result of a calculation) needs to be available in the language resource file(s). When there's text in the code that gets output directly (even if it's just a commas, parentheses, or spaces), it can complicate the translation job for at least some translators, and increase the likelihood of requests that the code be changed to accommodate a specific translation.

Hardcoded syntax:*

This one is harder to summarize, because each instance of hardcoded syntax occurs as a solution to a specific problem, and the thought process leading to that solution is different in each case. If we compare the two examples from esoTalk that I mentioned, for example, we find:

Toby's handling of the John, Paul, George, and Ringo situation was a very elegant solution to the problem of inserting an and before the last item in a list. When trying to solve to such a problem, one generally doesn't stop to consider that other languages may put the and in a different place. (And no wonder! Even though I've been speaking Japanese for decades, putting the and after the first item in the list still seems really odd to me.)

By contrast, the handling of the "Sort By Name/Posts/Last Active" button is a typical case of having one bit of text that remains the same and another that can change. Handling each of those bits as a separate string is a very common way to avoid creating multiple strings that partially duplicate each other. Again, this is an elegant way to handle the situation ... so long as you're not trying to translate it into a language with a syntax that's very different from English.

Thus we have two completely different but elegant solutions to two different problems, each arrived at through a different sort of thought process. It's hard to come up with a single way of handling all such cases.

So I guess the TL;DR here would be something like this:

When looking for solutions to problems involving how information is ordered for display, it's a good idea to add a sanity check at the end. Does the solution take a single sentence (or clause, or phrase) and chop it up into more than one string? If so, perhaps it would be better to find a solution that handles the entire sentence (or clause, or phrase) as a single string. The end result may seem less elegant, but when you take the long view, what's elegant for one language isn't necessarily elegant for all languages.

That's about as close as I can come, without knowing a lot more about programming than I do. I hope it can be of use.

Dominion

Stumbling block 3: too many duplicate strings

Duplicate language strings can make a translators' job more difficult, for fairly obvious reasons. First, more strings means more work. If a developer can do anything to reduce the amount of language resources involved in a project, that's less work that translators need to do. Even one less string may seem like a small amount, but that gets multiplied by the number of translators working on the project...

A good example of too many duplicate strings can be found in MyBB. That script puts the body text for each email template in a single thread, which isn't really a problem per se. Unfortunately, MyBB also allows the admin to configure the login method (which, don't get me wrong, I think is a wonderful idea!) and tweaks the information included in registration and address authentication mailings based on the specified login method. As a result, there are several longish strings that are all the same except for one crucial piece of information. That's a fair amount of extra work that each and every translator has to do.

Such duplication can also make it harder to maintain translations. If a translator should realize, several months down the road, that he has mistranslated a word, he could easily go back and edit just one of the strings, and forget about the other near-duplicates. The result is a language pack of unreliable quality.

Fortunately, @Toby has chosen to handle email strings for esoTalk in a more efficient manner: each mail template is broken down into its component strings (some of which are used in more than one mail), and the script assembles the body text by grabbing the necessary strings each time it sends a new mail out. I really appreciated how easy it was to translate the emails in esoTalk!

In fact, having worked a bit with both FluxBB and esoTalk, I get the impression that both @Franz and @Toby prefer to keep the number of language strings down to a minimum; so I'm really not too worried about seeing too many duplicate strings in Flarum.

So why bring it up at all? Because this issue is related to the next stumbling block...

...which I'll get to after lunch. 🙂

Dominion

One lunch later...

Stumbling block 4: too few duplicate strings?!

Keeping the number of language strings to a minimum is a great idea! Except it ignores an important fact about how language works, namely: just because one language has an expression that can be used in two or more different contexts, one shouldn't assume that another language will have an equivalent expression that can be used in both of those contexts. For example, we can use the English expression "That's cool!" when we think the temperature of a thing is relatively low, and also when we think something is neat, keen, far out, or totally rad. In most other languages, it would be hard to find a single word that would do both of these jobs.

Now, that's not an expression a translator of Flarum is likely to come across. But let's assume, for the sake of argument, that @Franz and @Toby decide to use this expression in a couple different places, each time with a different meaning. It won't be long before a translator comes along and says: "Um, we really need this to be two strings instead of one."

When you think about this sort of thing, it quickly becomes clear that there's only one way to ensure that translators will be able to get their job done without having to ask for new strings. Every time you reuse a string, you have to compare that instance against every other place where the string is used, and make sure that the new instance is being used with the same meaning, and in the same context, as all the others. Because any slight variation in meaning or context could be justification for creating a new string. Unfortunately, that's really hard to do while you're busy writing a program!

So we should probably fall back to the next best option, which is to identify some common situations where the same expression may end up getting used with different meanings, or in different contexts, and provide for those situations. As an example of this, let's look at a hypothetical situation involving three different uses of the same expression.

Imagine you're looking at the Flarum UI and you see a link that reads: Change Password. When you click this link, it opens a page or dialog box that is titled: Change Password. Below that are text boxes where you can type in your new password, and a button that also reads: Change Password.

... When I called this a hypothetical situation, I was talking about the specific details. Speaking more generally, this is the kind of thing that happens in just about any program. In this case we've got the same pair of words being used in three different contexts; and because of the contextual difference, the meaning (or intention, if you prefer) is slightly different in each case.

The link puts words in the user's mouth: "I'm clicking this link because I want to change my password."
The title is the script telling the user what he's seeing: "This is where you can change your password."
The button speaks for the user again: "Okay, I'm all done typing. Please change my password NOW!"

It just so happens that in English (and perhaps in many other languages) these three different shades of meaning can be conveyed by the same two words: "Change Password". In other languages, however, trying to use the same string in all three places can cause the translator some difficulties. Let's look at Japanese as an example.

I could probably find a way to use the same expression in all three places, but the result would be a rather telegraphic Japanese mostly suited to a casual site for computer geeks. If I wanted to make the site more fluent (for less computer-savvy users of a corporate site, say) then I'd prefer to do something like this:

For the link: pasuwādo-wo henkō suru
For the title: pasuwādo-no henkō
For the button: henkō suru!

... Where henkō is a noun meaning "change" and suru makes it a verb; the -wo marks the noun pasuwādo as the object of the verb, and the -no is the possessive. Thus, the above could be translated roughly as:

For the link: (I want to) Change my password.
For the title: The Changing of the Password
For the button: Change it!

Of course I'd be using only the verb in the button to avoid having a really lo-o-ong button; I can get away with it because the noun pasuwādo is already there in the title of the dialog box. Now, if I didn't have any aesthetic objection to buttons that stretch a third of the way across the dialog box, I could get away with using the same expression for 1 and 3. But the difference between 1 and 2 is crucial, as it matches the language to the context. Even if I wanted to omit the final suru from 1 (which is possible in Japanese, since the presence of the -wo indicates that the following henkō should be read as a verb), I'd still need two separate strings to preserve the difference between the -wo and the -no.

To those who didn't really want a crash course in Japanese grammar, I apologize for the technical stuff. The main point I'm trying to get across is this: there are indeed times when duplicate strings may be needed to translate what seems like a single expression. In such cases, forcing the translator to make do with one string makes it harder to adapt the translation to specific user needs. in other words, it decreases the workload for everybody, but limits the freedom of, and makes the job harder for, a few.

Here I should probably mention that, in my experience, developers rarely use duplicate strings in a really consistent manner. Even in situations like the hypothetical one described above, we sometimes find the same string for 1 and 2, but a different string for 3; or the same string for 1 and 3, and a different string for 2. Even in the same project, it's not uncommon to find a mixture of these and other approaches, depending (I suppose) on what the programmer felt like doing at the time.

Now, I don't know how @Toby and @Franz feel about the idea of adopting some sort of HID standard to ensure that language is used consistently in things like links, dialog titles, and buttons, but ... If you guys are considering that sort of thing, that might also be a good place to think about adding duplicate strings that translators may need.

I'm hesitant to make any specific recommendations along these lines, because without looking closely at how language is being used in the Flarum UI, it's hard to say exactly where such provisions may be needed. Also there's a bit of a trade-off involved: every time you add a duplicate string to provide for potential differences in the handling of a certain expression, you're increasing the number of strings that all translators will have to translate. So leveling the field for some translators will mean more work for all.

I can say this much, though: if you do include duplicate strings to cover situations such as I've described, it's important to provide clear hints (in the form of understandable string names or comments, for example) as to which string goes where. Without that information, translators can spend a lot of time trying to get the right translation in the right place. Beyond that, I can only say that this is a subject that may require more discussion once translators have had a chance to play with the beta.

(I'm also wondering if YAML might not offer some way to take a single string and translate it as several strings, depending on the context. I'll have to look more closely at how variants are conditioned.)

Dominion

Since I was just writing about the importance of giving clues as to what string goes where, here's a topic that may offer a solution to that problem:

Stumbling block 5: organization of language resources

Oddly enough, how the language resources for a project are organized can turn out to be one of the biggest stumbling blocks for translators. When resources are badly organized, or organized according to principles that aren't followed scrupulously, translators can spend a lot of time just trying to find a string they want to translate. The whole job takes longer, and the job is less rewarding.

By and large, organizing strings by file or module seems to be the most popular method. This sounds like a good idea, but more often than not it doesn't really work out. Especially when there are many volunteers contributing to a project, it seems little effort is made to maintain policies regarding what strings go where. The result is, well, a somewhat organic arrangement. There's the "global" file that ends up being a huge catch-all, and inevitably includes a few strings that are only ever used in one place. And there are usually a few strings that are used in more than one file, but don't get put in the "global" file, so it's anyone's guess as to which file they can be found in.

(Hint: An organic arrangement is a neat concept if you're an artist, but not so good for technical things like programs.)

Translation GUIs could be a help to translators, except they also tend to be organized by file or module, so translators still have to hunt around a bit. And since that hunting involves loading files over a network, the translation GUI tends to be more hindrance than help. A translation GUI without a global search function really isn't worth the trouble. (That said, I think the best translation GUI is the one in MyBB, which at least allows you to submit multiple strings at once. Many other scripts force translators to submit strings one at a time, which can result in a huge waste of time.)

Having tried translating a few forum scripts, I think esoTalk has provided the best experience. This is mainly because @Toby put nearly all the strings in a single file. At first I thought this would make life harder, but it turned out to be much easier, simply because there's no hunting around for strings and no need to muck about with multi-file searches. (My editor is a bit clunky when it comes to searching multiple files.) Also, while there's no translation GUI, editing the resource file is dead simple as long as you're careful not to delete the single quotes that delimit the string.

The only potential issue I see in the method used by Toby is this: lumping all the strings in a single file can increase the possibility that a single string will end up used in more than one context, making it necessary to duplicate the string to support certain translations. (See the post above this one.) But that's not much of an issue. If the strings were separated into several files, there's a good chance that the same string would end up in the "global" file anyway.

As for how strings were organized within the file, I was surprised to see that Toby had filed them alphabetically. But this turned out to be very useful, since it made it possible to find strings quickly without using the search function. That seems like a schema worth sticking to. There is also a "Messages" category (identified by a comment) that makes a lot of sense; so much so, that I think it might be worth looking into the possibility of using the idea of categories more widely.

Let's say, for example, that it becomes necessary to provide duplicate versions of strings as described above: one version of "Change Password" for the link, one version for the dialog title, and one for the button. And let's say that in order to provide for consistent use of language, we're going to include duplicate strings for every situation where there's a link, a dialog box, and a button. Categories similar to the "Messages" category that Toby provided for esoTalk could be used to keep these strings separate from each other.

That is, in addition to the "Messages" category there would also be a "Links" category containing all strings that are rendered as links to other pages or dialogs, a "Dialog titles" category containing the titles for all dialog boxes, and a "Buttons" category with the text for all buttons. Any strings that don't fit in these categories would go in the general alphabetized section.

That's just an example of something that could be done, not a recommendation of something that should be done at this point. (After all, it really depends on how the matter of "too few duplicate strings" will be handled, among other things.) I've never seen this sort of categorical organization used extensively in any script I've tried translating, but it seems to me that it could be much more useful than organizing strings based on the file or module where they're used.

Toby

@Dominion This is an absolute gold mine of information, not to mention it's actually been quite easy and entertaining to read. I love your writing style. Splendid work!

To reply to each block briefly:

Stumbling block 1: hardcoded text

No arguments here. We'll do our best to hardcode nothing, but inevitably we will probably miss some spots – so please don't hesitate to speak up when you notice them after you start translating!

Stumbling block 2: hardcoded syntax

As above. We fully intend to make everything you've mentioned as flexible as it needs to be to produce quality translations. You'll be pleased to know that the arguments in translation strings are named, so instead of posted by %s at %s you'll be working with posted by {username} at {time}, and can rearrange them as you please.

Stumbling block 3: too many duplicate strings

Yes, absolutely. We have no intentions to create unnecessary work for translators. 🙂

Stumbling block 4: too few duplicate strings?!

This is a very good point which I've not considered before, so thank you for bringing it up. My preferred solution here is to suffix variations of the same key for different contexts. For example:

change_password_link: pasuwādo-wo henkō suru
change_password_title: pasuwādo-no henkō
change_password_action: henkō suru!

This way all of the relevant translations remain grouped together, but the variations are still distinguished clearly.

(As you mentioned, the semantics of link and action are almost identical, and the difference is the length of the translations with respect to where they'll be displayed. I feel like splitting them up like this lacks a bit of elegance, though – purely from a length point of view, I think the longer _link translation would look fine in a button... or perhaps, could the button just use a generic "save changes" translation?)

Stumbling block 5: organization of language resources

Due to the nature of Flarum's extensibility, translations will be grouped by extension. So all of the core translations will be in one file, and all translations for the Tags extension will be in another, etc. Within each file, they will be listed alphabetically. So, very similar to how it's done in esoTalk really.

Translations will be further grouped by prefix. A few examples:

notification_method_alert: Alert
notification_method_email: Email
group_admin: Admin
group_admins: Admins
sort_latest: Latest
sort_top: Top
delete_account: Delete Account
delete_account_help: "Hold up! If you delete your account, there's no going back. Keep in mind:"

I think a good approach is to keep the translation keys:

Generic so that they are not tightly coupled to the English translation (e.g. the translation key for "Enter a new title for this discussion:" should be prompt_rename_discussion instead of enter_new_discussion_title)
Descriptive of context, as we've talked about a bit already (e.g. for the "discussion renamed" notification, which is "{username} changed the title", the key should be discussion_renamed_notification not discussion_title_changed)

You might be interested to take a look at the English translation file for Flarum's core in its current state. I did this the other day, before I read a lot of your posts, so it's far from perfect. I probably won't be able to tweak the keys in time for beta launch, but certainly afterwards we can collaboratively improve it!

Allineer

Toby Translations will be further grouped by prefix.

I propose to separate the key form the base suffix with two underscores. For example:
```
change_passwordlink: pasuwādo-wo henkō suru
change_passwordtitle: pasuwādo-no henkō
change_password__action: henkō suru!
```

```
notification_methodalert: Alert
notification_methodemail: Email
groupadmin: Admin
groupadmins: Admins
sortlatest: Latest
sorttop: Top
delete_account: Delete Account
delete_account__help: "Hold up! If you delete your account, there's no going back. Keep in mind:"
```
I think this will further visually grouping of prefixes.

PS: @Dominion, you are amazing! 🙂

Dominion

Toby This is an absolute gold mine of information, not to mention it's actually been quite easy and entertaining to read. I love your writing style. Splendid work!

Thanks Toby! I know I have a tendency to run on a bit, so the least I can do is try to keep it interesting. 😛

Toby so please don't hesitate to speak up when you notice them after you start translating!

Will do! o7

Toby so instead of posted by %s at %s you'll be working with posted by {username} at {time}, and can rearrange them as you please.

Wonderful!

Toby This is a very good point which I've not considered before, so thank you for bringing it up.

I'm glad you found it useful. It's the point I was the most concerned about, since lack of flexibility can really make or break the quality of a translation.

Toby My preferred solution here is to suffix variations of the same key for different contexts.

Hmm ... that's quite different from my later suggestion of using categories to group similar objects (links, buttons, etc.) together. But I think your way might be better in the long run. Since the items being grouped together will appear on the same page (or at least relate to the same flow of operations) having them grouped together will make translating faster and easier.

Toby purely from a length point of view, I think the longer _link translation would look fine in a button

That may be true in many cases. But although Japanese translations tend to require fewer characters than the English originals, the characters are a bit larger. And when you start involving transliterated words (like pasuwādo), which are generally longer than native Japanese words, that button can get pretty big. I think ultimately it would be nice to have the option of keeping the button name short. (I wonder if @Franz wouldn't have a similar opinion, since I know some German words can also get rather long!)

Toby or perhaps, could the button just use a generic "save changes" translation?

In fact, many Japanese programs just use the word sōshin (which means simply "Send" or "Submit") for such buttons (everything from "Save Changes" to "Change Password" to "Post Reply"). So that's probably the best way to go. Not only will it allow for less obtrusive buttons, but it will also cut down the total number of strings needed.

Toby Due to the nature of Flarum's extensibility, translations will be grouped by extension... So, very similar to how it's done in esoTalk really.

That sounds perfect.

Toby Generic so that they are not tightly coupled to the English translation (e.g. the translation key for "Enter a new title for this discussion:" should be prompt_rename_discussion instead of enter_new_discussion_title

I agree. I've always found keys like "enter_new_discussion_title" to be fairly unhelpful. The really important thing is to come up with a rational plan and then stick with it. (But shouldn't it be "rename_discussion_prompt" if you'll be suffixing?)

Toby You might be interested to take a look at the English translation file for Flarum's core in its current state.

I'll grab a copy and have a look at it, but I think I might have to actually start translating to provide any real feedback. And that book translation I mentioned is starting to breathe down my neck!

EDIT: ... Only 115 lines so far?! Whee! Hooray for the minimalist approach!

Toby I probably won't be able to tweak the keys in time for beta launch, but certainly afterwards we can collaboratively improve it!

I look forward to helping out in whatever way I can!

Dominion

Toby jumped in before I was quite done. Then I was called away for dinner. And so now, newly refortified by a big bowl of bibimbap, I give you the final chapter (ta-dah!):

Stumbling block 6: content-related issues

... which isn't really a stumbling block, so much as a grab-bag of things that caused me a bit of trouble (or at least made me scratch my head) while translating esoTalk. I suspect that some of these may already be non-issues in Flarum, but it might be worth jotting my comments down here in case they'll come in handy while planning new features or extensions.

URL oddness

I was pleased that esoTalk had very few problems turning Japanese thread titles into pretty URLs. But there was a bit of oddness that puzzled me. For example, the URL for a thread titled

この「ＤＣＳ」会議室を使う前に読んでください！

which means, "Read this before using this 'DCS' conference room!" would end up looking something like this:

https://www.example.com/93-この-ｄｃｓ-会議室を使う前に読んでください

It seems the Japanese in the URL has been adjusted by some processing intended to render it compatible with the ASCII standard. The 「 and 」 (which are quotation marks, incidentally) have been replaced by hyphens, and the final full-width exclamation point has been dropped. Also the capital letters have been made lower-case, even though they remain full-width and so are not ASCII compliant. All in all, it's a fascinating display of how processing intended for Latin strings can have the intended effect on Unicode. And it's not a huge problem, since the URLs are still pretty readable. But since we're talking about a Japanese URL in any case, I can't help but think: "Why bother?"

If current plans call for the inclusion of similar URL processing in Flarum (as appears to be the case), perhaps it would be a good idea to allow admins to turn it off when it's not needed? Of course, the same would apply to channel slugs (er, I mean tags): it would be nice to be able to use capital full-width letters there as well.

Search function

This isn't exactly a "content-related" issue, but...

I had fun trying to translate all the search gambits properly, and think I more or less succeeded. But because some of the gambits rely on non-Japanese glyphs like half-width colons, spaces, and question marks to determine how much of the text is selected by default, the results aren't quite as polished as I would have liked. Also, it was getting a bit hard to avoid Yoda-speak on the more complex gambits, mainly due to the need to pack a lot of information into a small space.

Since Flarum won't be able to search for text strings in Japanese (not to mention Chinese, Korean, etc.) unless the admin can supply the necessary infrastructure (which will require, at minimum, a VPS or dedicated server with root privileges), the ability to search using a combination of other criteria will be especially important for applications on shared or managed hosts. That being the case, I'd like the advanced search function to be as polished and useful as possible.

I don't how the plans for advanced searching in Flarum currently stand, but I've been wondering if it wouldn't be better to go with a more conventional "advanced search" page with the various criteria laid out in some slick fashion. That would pose less of a challenge for translators; and ultimately, I think users may find it easier that way to keep track of which criteria they've turned on for a specific search.

Alphabetical scrubber

Flarum doesn't seem to have a member list page yet, so I can't be sure whether you're planning on adapting this feature from esoTalk. But just in case you are, here's a few thoughts on the subject:

Generally speaking I think it's a cool feature and a spiffy bit of code; but it's pretty much wasted on a forum whose members will mostly have user names in non-Latin characters. So ... I don't know if it'll make much difference, but if you're going to include the alphabetical scrubber, perhaps it would be a good idea to add an option to turn it off? On a forum where hardly any names are alphabetize-able, it would only be in the way.

The alternative would be to get fancy and make the scrubber itself localizable. The language pack could include a "namescrubber" file that would list the characters to be used by the scrubber, in the proper order. When you install the language pack, poof! The scrubber changes. But to make this work, there's a couple obstacles that need to be dealt with.

First, you wouldn't want the name scrubber to change depending on the user's language setting. If most of the users' names are in Japanese, for example, then a user who changed the UI language to English would be rendering their scrubber unusable. So you'd need to include a "scrubber language" setting in the admin options, which would set it for all users, period.

Second, while the "name scrubber" file might be enough to handle most languages, there are some languages that will need even more. Japanese is one of those. I'll go into a bit of detail in the next item.

Username sorting

Japanese user names can include characters from either of two syllabaries (of around 50 kana each) and a couple thousand or so Chinese characters. And while it's possible to do some processing and sort names using just the kana, the pronunciation of all those Chinese characters isn't fixed: most have at least two pronunciations, and many have several possibilities. So there's no easy way for a program to look at a Japanese name and decide how it is pronounced, or how to sort it with other names.

Most Japanese websites get around this problem by adding a "reading" field, which is restricted to characters from one of the two syllabaries. Users enter the reading of their name in this field, and this data allows the program to sort their name correctly.

So in order to enable localization of the name scrubber for Japanese, we would need to give admins some way to define custom profile fields, including a way to constrain a field so it only allows characters from a specified syllabary, as well as a way to specify that a given custom field is to be used instead of the username as the sort criterion. All told, that's a pretty tall order. In fact, in consideration of this, I'm beginning to think it might be best to put the name scrubber in an optional extension.

But even if we go that route, we still need to consider what happens to the member list when sorting is left up to the database. We can be pretty certain that many names will be out of the expected order. If the forum user base is large, names will be hard to find. A good username search function is clearly going to be needed, but it seems some way of specifying how a name is read would probably be a good idea in any case. Is this the sort of thing that can be done with an extension? And what is the relationship of the extension to the language pack? Would it be part of the language pack? or a recommended add-on?

This is a peculiarly Japanese issue, but I'd guess there are at least a few languages out there with similar issues.

Localization of editor buttons

Okay, this is not a major issue, but ... while translating esoTalk, I found myself wondering if it would be possible to replace a couple of the editor buttons with more Japanese alternatives. For example, it might be nice to have a bold 太 ("fat") character instead of a B, or an italic 斜 ("slanted") character instead of an I. And so on.

This isn't absolutely necessary, since Japanese users have become accustomed lately to seeing the standard buttons, but I thought it might be a slick touch to add. Would it be possible to create such buttons and package them with a language pack?

Of course, it would be necessary to provide some way for the buttons to change when the user changes the UI language setting. And there are interactions with alternate markup methods to consider. On the other hand, localized editor buttons might be a nice addition in more languages than Japanese. Just a thought ...

Dominion

Allineer I propose to separate the key form the base suffix with two underscores.

I'm not sure that's entirely necessary, but it could certainly help make suffixes recognizable as such.

Allineer PS: Dominion, you are amazing! 🙂

Aww, shucks... (#^{_^#)}

I just thought it would be nice to give the translator's view ahead of time for once, rather than grumbling afterward... ;-)

Franz

This article may also be a good resource on this topic.

Dominion

Franz This article may also be a good resource on this topic.

Thanks ... I actually read that a week or so ago, when I was contemplating starting this thread. One big difficulty in Flarum's case is that it's hard to plan ahead for specific languages, since the goal (I've been assuming) is to open the project up to as many languages as possible. So while I've spent a lot of time talking about Japanese, I'm really hoping that my comments will help you guys accommodate as many languages and translators as possible. 🙂

Toby

Dominion

URL oddness

I'm a bit confused – I just tried creating a conversation in esoTalk with that title, and the URL ended up being http://esotalk.org/forum/1993-dcs ?

Anyway, to be honest I haven't done much research or thinking on the best way to go about slug generation – currently Flarum simply strips all non-alphanumeric characters, which is obviously not ideal for non-Latin languages. This is something we'll need to nut out eventually...

Search

I'm thinking the best way might be to make gambits non-translatable, and then as you suggested, have some kind of "advanced search" user interface to abstract the gambits away. Gambit translation was a big kettle of fish in esoTalk, and even then it was still far from perfect.

Alphabetical scrubber & username sorting

A member list page is something we'll get to eventually (maybe in core, maybe an extension – not sure yet). My plan is to make it a bit different to esoTalk. Instead of sorting alphabetically, it will be sortable by various activity metrics. There will be no letter scrubber. So I think that nullifies both of those problems?

Editor buttons

I think that would be possible to do, but it will require some code added to the language pack.

Cheun

Holy. This is the longest thing I've ever read and didn't get tired of reading ( and I hate reading ). Really learned a lot here.

Dominion

Glad you got something out of it!

Heck, when I started out, I didn't think it was going to get this long...

Franz

Toby Alphabetical scrubber & username sorting

A member list page is something we'll get to eventually (maybe in core, maybe an extension – not sure yet). My plan is to make it a bit different to esoTalk. Instead of sorting alphabetically, it will be sortable by various activity metrics. There will be no letter scrubber. So I think that nullifies both of those problems?

Reminds me of StackOverflow. They have some cool user ranking pages...

Dominion

Toby Thanks for the feedback!

Toby I just tried creating a conversation in esoTalk with that title, and the URL ended up being http://esotalk.org/forum/1993-dcs ?

Interesting ... perhaps it has something to do with region settings? It works all right on my site.

Toby This is something we'll need to nut out eventually...

I'll be glad to offer whatever help I can.

Toby I'm thinking the best way might be to make gambits non-translatable, and then as you suggested, have some kind of "advanced search" user interface to abstract the gambits away.

I don't see any trouble with the actual gambits staying in English as long as there's an easy-to-use interface; so that sounds like the perfect approach.

Toby My plan is to make it a bit different to esoTalk. Instead of sorting alphabetically, it will be sortable by various activity metrics. There will be no letter scrubber. So I think that nullifies both of those problems?

It does. Sorting names by activity metrics sounds great! But I've been assuming that an alphabetical (or however names are normally sorted) list would be needed for those cases when somebody just wants to look up another user quickly. Some sort of username search function can be useful for that (I seem to recall FluxBB had a nice one), but only if you can remember how the username is spelled.

Am I overlooking something really obvious here? Is there another tool or strategy besides alphabetization (or whatever) that can be used to locate a username quickly?

Toby Editor buttons

I think that would be possible to do, but it will require some code added to the language pack.

Cool! This is one of those things that aren't at all necessary, but would be a really nice touch to add.