Text formatting in Flarum

Glad you like it.

If we forget about storage space, what really is the benefit of TextFormatter's strategy over this one?

For one, you don't have to invalidate your whole cache when a template is changed. Some general examples, not all may apply to Flarum specifically:

• Adding a class name to links
• Changing the markup for an emoticon, or the path to their images
• Whenever Twitch will decide to ditch their embeddable Flash player for a `` tag

It's also more dynamic. You can more easily localise some things like dates, or use variables in templates. For example, in the XenForo add-on I was talking about people can post a link to an Amazon product and it's displayed as an Amazon widget. If they become Amazon Associates, they can enter their affiliate ID and the widgets will earn them revenue. The ID is stored in a template variable so when they add an affiliate ID, it applies to old posts as well as future ones.

In phpBB, template variables are used for localised strings, and to apply some user preferences such as toggling images, emoticons or Flash applets. They can also be used to display things differently whether the current user is logged in ("Hide from guests" extensions are a common user request in every forum) whether it's a bot or depending on the user's style preference.

As for storage, the extra space is only one aspect of the storage requirements. DB storage is relatively cheap these days so it's rarely an issue. I'm more interested in the extra column(s) or table(s) an HTML cache would need, especially if you ever want to cache content from more than posts. Perhaps you want to allow users to leave each others messages on their profile or have some sort of Twitter-like micro blogging and if it's not built using the same structure as posts then you may end up with multiple HTML caches.

If a formatting extension is enabled/disabled (say, BBCode is turned on/off), wouldn't that affect the structure of the XML – and thus, wouldn't you have to unparse and then reparse every post in order to update output?

If you want to retroactively remove BBCodes from old posts while enabling them in future posts then yes, you'd have to reparse everything. Otherwise, you could just replace every template used for BBCodes with something that contains only text.

One other minor concern is that it tightly couples all post content to a library. Storing posts as XML kind of obfuscates their content in the database, so if you want to go in and manually edit things, it's a lot harder. If people want to export their data, or migrate to another forum, it makes it harder.

The very first decision I took about the library, before I wrote the first line of PHP, was about the storage format. It's XML and if you load it in any XML parser/DOMDocument and retrieve the textContent of its root node, you end up with the original message. The original message can be retrieved in any language with little effort and no knowledge of the format.

```xml
Hello world
```

As far as multiple formatting styles/markup languages go, I rather like that once it's been posted, the structure of a message isn't effected by changes in the supported markup languages. For example, Textile and Markdown have an incompatible syntax for emphasis/italic vs strong emphasis/bold, or for headings vs enumerated lists. If the admin switches off Markdown for Textile then old posts could look completely different after reparsing.

Realistically, I don't think admins change their markup language very often though.

Parse all of the formatting styles in core, but only render them if the appropriate extension is enabled.

I don't understand how that would work. For example, in Textile ## is an enumerated list but in Markdown it's a heading. It's the user's choice to decide which one it should be. What you could have is an option to choose markup languages by default and/or at posting time.

Still, this would limit the possibilities of what extensions can do, without having to...

It does but it's a blessing and a curse. For example, let's say someone installs an extension that turns YouTube links into videos. Old posts will still display links, not videos. It sucks for short posts that say "Hey! look at this video: http://youtube..." but in other cases it can be preferable. For example, someone may have posted the list of the 100 dankest YouTube memes as a list of 100 URLs and if they turn into embedded videos overnight the thread may become unusable because of the 100 iframes it would contain.

How does something like the censor plugin work, where the words that are being formatted could be changed quite frequently? Would that require a global reparse?

The Censor plugin works two ways, as a normal plugin or as a standalone and you can use either or both. The normal plugin applies at parsing time; changing which words are censored does not effect old posts. The standalone is an helper class that's completely separate object that can be used at rendering time.

I don't have a strong opinion on what's the best approach there. Censoring at rendering time incurs a small performance penalty, proportionate to the complexity of the list of censored words. Censoring at parsing time means it costs no resources at rendering time and it can still be toggled (via a template parameter/conditional) to display the text censored or uncensored. On the other hand, if you add a word to the list, that's one case where you'll want it to be retroactive and apply to old posts. On that note, the helper class has a method to quickly update what's censored in an old post without touching any of the markup.

Another question I forgot to ask before: Does this library have any dependencies? We want to make sure Flarum is installable on low-end shared hosts.

No dependencies.

Oh, one more: Are you going to do SemVer? We need something we can rely on to not break BC.

I'm still working on my release process. Up till now I didn't need a version number so I never stamped one on the sources. SemVer could be the way to go.

I always thought your intermediate XML syntax is so generic that it can be converted into more or less any of the possible markup languages (by unparsing). Is that true?

Unparsing always returns the original message. The XML could be used to transform the original message from one markup language to another, but that's not a current feature. It's been on my TODO list for years but I haven't prioritized it for lack of application.

A user could select whichever markup language but switching it would not change [b]bold[/b] into **bold** or *bold* in the original text.

I don't remember writing any doc about custom plugins. One of the reason is that every time I started thinking about an example of a custom plugin, the use case was better handled using an existing plugin. I wrote a page about custom parsers if that's any help. A plugin is a parser + a configurator.

I don't know how @replies#123 is supposed to work, can you break it down for me please? I assume that 123 is the ID of the post, but what's the replies part?

About @mentions, the simplest way would be to use the Preg plugin to match them. Then you have to retrieve the user ID. The simplest way is to use a tag filter to add the user ID to each tag during parsing. That means it'll perform one query per username mentioned. If you're concerned about a malicious user mentioning thousands of usernames, you can set a limit to the number of mentions in a post. Here's a working example that matches /@(?\w+)/ and creates a tag named MENTION for each mention.

If you really want to run only one query to retrieve all user IDs, you could preload user IDs before parsing, or implement your own parser that would preload all IDs before creating any tags. The downside is that it would query IDs for things that look like mentions but are not. For example, things in inline code such as @mention.

Alternatively, you could not add any ID during parsing and inject them after parsing in the XML. You wouldn't query IDs for inline code and such. On the other hand, it's more complicated.