Need Community Input on Non-English Character Support

tankerkiller125

Hello Flarumites,

Here at Flarum we want to be inclusive to everyone regardless of where they live or the language they speak. We understand that Flarum is being used all over the globe (something we love) but in some cases our support for languages other than English is lacking.

This is why we are asking for your feedback. We plan to address these shortcomings over time but we need to know what all of them are. We have identified the following issues so far:

For the uninitiated, UTF-8 in this context is a character encoding standard that supports the use of many different language characters from all regions of the world as well as Emoji. Essentially, when we say UTF-8, we mean characters in non-English languages.

UTF-8 Usernames are unsupported (flarum/core1949 and flarum/core1990)
- Mentions currently can't match UTF-8
- User slugs are currently hard coded to be the username
Discussion slugs can not be UTF-8 and transliteration does not work for some languages/cultures (CJK) (flarum/core194)
UTF-8 search is awful (flarum/core2003)

You can keep track of our progress on these issues on flarum/core2230

If you have other issues that you've found regarding languages not working properly in Flarum please let us know in this discussion (preferably with some examples) so that we can test and add them to our list if needed.

tankerkiller125

danielunited This falls under

tankerkiller125 UTF-8 search is awful (flarum/core#2003)

I believe, we have discussed some ways to resolve this by allowing search drivers for services like Elasticsearch, Algolia, etc. as MySQL/MariaDB is not good at UTF-8 search regardless of how you configure it.

meetdilip

The URL skips the regional language part ( from title ? )

I have not tried the 3 points you mentioned though. Content and title works for me right now.

tankerkiller125

meetdilip I believe that this falls under point two if your referring to the fact that a title in say russian cyrillic won't be converted into a proper slug but stead becomes a bunch of dashes (if I remember how that ends up right) but any english in that title will be correct.

meetdilip

tankerkiller125 a title in say russian cyrillic won't be converted into a proper slug but stead becomes a bunch of dashes

It does not create a bunch of dashes, but a simple, proper alpha-numeric URL like in WordPress.

tankerkiller125

meetdilip Does it currently do that? Or is that what your hoping it will do? I'm trying to figure out exactly what you mean by "regional language part" in your original reply to the discussion.

meetdilip

I use a non English language title. Instead of a SEO URL that we get with an English title, it shows

https://domain.cloud/d/55

all by itself.

tankerkiller125 Does it currently do that?

Yes. It automatically skips the non English part.

tankerkiller125 Or is that what your hoping it will do?

It is working like that now

bryantmilan

Well, I found one small issue while I was translating from english to serbian language. In the serbian language, we have a difference between the masculine and the feminine. In fact, some words do not have the same ending depending on gender. Here's an example to better understand:

For the masculine:

English: John started the discussion a few seconds ago.
Serbian: John je započeo diskusiju pre nekoliko sekundi.

For the feminine:

English: Barbara started the discussion a few seconds ago.
Serbian: Barbara je započela diskusiju pre nekoliko sekundi.

As you can see there is a difference between the words "započeo" and "započela". When I was translating from english to serbian I had to use the universal word "započeo/la" so users could use correct form of the word depending of the situation. It would be great if during registration users could choose what gender they are, whether they are male or female. The translation would be adjusted accordingly. This would make the translation much more meaningful and make it easier for users to use and read on forum. I am not an expert in literature and grammar, but I think this does not only apply to the serbian language but to all the languages of the former Yugoslavia and I think it also applies to the russian language, but I do not want to speak on their behalf. This isn’t such a big deal, but it would be great to have.

clarkwinkelmann

bryantmilan it's not strictly related to this thread's topic as it's not related to character encoding. We do have an existing issue for genders but it's a very complex issue as we need both the translator library to support genders but also some way to know the gender which is quite delicate. Feel free to give input on the GitHub issue if you have experience with this kind of translation feature flarum/core528

danielunited

The search functionality doesn't work well on my Hebrew communities.

It doesn't search parts of the title (for example if you type in just 1 word from the title).

danielunited

I also notice as issue with SEO. It displays the date of each comment and its URL in search results.

tankerkiller125

danielunited I'm not sure if this is a UTF-8 related thing or a regular forum thing.... I personally don't see that issue happening on Google with discuss, maybe someone else with a international forum can confirm?

Edit: Further testing, I put the URL of the forum you showed the issue on into google and I do not see that happening. Mind sharing which search engine you see this on?

matteocontrini

tankerkiller125 I can't reproduce either, both on Google desktop and mobile

ceerker

Maybe you can refer to this：https://github.com/Csineneo/flarum-hack/blob/master/hack.sh

ceerker

Client language recognition

wget -qO "vendor/flarum/core/src/Locale/LocaleServiceProvider.php" \
"$GITHUB_ROOT/flarum/core/src/Locale/LocaleServiceProvider.php"

Allow registration of Chinese name

sed -i "s#a-z0-9-#-a-z0-9\\x7f-\\xff#" \
vendor/flarum/core/src/User/UserValidator.php

Support @ Chinese name

sed -i "s#a-z0-9-#-a-zA-Z0-9\\x7f-\\xff#" \
vendor/flarum/mentions/src/ConfigureMentions.php

Cancel the minimum length of username

sed -i's#min:3#min:1#' \
vendor/flarum/core/src/User/UserValidator.php \

Allow searching for IDs shorter than three characters

sed -i's#length>=3\&#length>=1\&#' \
vendor/flarum/core/js/dist/forum.js

Improve support for Chinese urls (fix some Chinese nickname user page analysis issues)

sed -i's#getIdForUsername($id)#getIdForUsername(urldecode($id))#' \
vendor/flarum/core/src/Api/Controller/ShowUserController.php
To

url only shows id

sed -i'/discussion->slug/d' \
vendor/flarum/core/src/Api/Serializer/BasicDiscussionSerializer.php
sed -i -r's#(discussion->id).$#\1#' \
vendor/flarum/core/views/frontend/content/index.blade.php
sed -i'/idWithSlug =/s/..$/;/' \
vendor/flarum/core/src/Forum/Content/Discussion.php
sed -i's#+(i.trim()?"-"+i:"")##' \
vendor/flarum/core/js/dist/forum.js

tankerkiller125

ceerker Although this does allow UTF-8 it is not extendable easily and does not give the forum owner complete control over how they want to slug things. I think our goal will to make minor changes to core that allow Extension developers to define slugging techniques and then allow the admin to choose from those options in the dashboard. Very similar to the way we handle email and "Front Page" settings.

However some of those things in the script do point out some issues we'll have to work on resolving (such as min search length)

wloot

tankerkiller125 About date format, this may helps: https://discuss.flarum.org.cn/d/1248/3

cccRaim

There is a solution for Chinese & Janpanese, but the title also say the solution is not perfectly.
https://discuss.flarum.org.cn/d/1216

tankerkiller125

cccRaim Thanks for your suggestion a similar discussion exist on this forum., the issue with the solution is that it requires modification of the MySQL default config. This is not something we can reasonably require/request all users/people to do and as such we won't be using this method. At some point we plan to have search drivers which would greatly help with the situation.