Fixing some shortcomings in litedown

Pollux

JoshyPHP First a bit of context: I'm the author of the text-formatting library used by Flarum. It handles all of the markup, including BBCodes and Markdown-style formatting, and enables Flarum extensions to add their own markup and features such as mentions or custom BBCodes. Changes to the core plugins automatically trickle down into Flarum. The plugin in charge of the Markdown syntax is named Litedown, and the name is used interchangeably to describe the plugin and the syntax of this Markdown flavour. The syntax itself is based off the original Markdown syntax and tries to support the most commonly requested/used extensions to the syntax.

I understand, that this thread is about spoilers. I want to ask nevertheless:

Are there any chances to fix some of the basic shortcomings like missing subset and underline tags in Markdown / Litedown (and no, BBCode is not an option for me). Would it be worth to start a separate discussion for that?

JoshyPHP

Pollux Sure, you can create a discussion and tag me in it. The first step would be to collect information about existing implementations and other similar discussions.

I don't know what you mean by "subset". If you meant subscript, it's described there.

jordanjay29

Pollux JoshyPHP split to a new discussion for you.

Pollux

I wrote about my headaches with superscript here nearly two years ago: Pollux

And to my surprise and pleasure, the two most annoying shortcomings have already been adressed with the latest iterations of the flarum software: Not beeing able to actively close the superscript tag and a missing subscript tag (yes, that's what I intended to say).

If I may present another wish: I don't understand how on earth the underscore (underline) character doesn't produce underlined text, i.e. _text_ doesn't produce text. It's counterintuitive imho., especially as emphasis is also covered by the asterisk.

By the way: Is there a reason for the different behaviour of _ and *? It seems, that single underscores don't work inside of words.

m*m*m >> mmm
m_m_m >> m_m_m
m**m**m >> mmm
m__m__m >> mmm

And one more unexpected behaviour: If you combine subscript and strike-through via triple tildes (~), a double tilde terminates both:

~~~m~~m~ >> _mm~

matteocontrini

Pollux If I may present another wish: I don't understand how on earth the underscore (underline) character doesn't produce underlined text, i.e. text doesn't produce text. It's counterintuitive imho., especially as emphasis is also covered by the asterisk.

Markdown does not have underlined text in general

JoshyPHP

Pollux I'm going to respond in the same order:

I'll look into it this week. The superscript behaviour (especially a^b^c) is mostly based off Reddit's behaviour at the time it was implemented. I might make the implementation aware of word boundaries based on ASCII punctuation but I want to think about it carefully. Many Markdown implementations base establish work boundaries off ASCII letters, which can cause issues with non-latin languages.

I wish that was the case as well but almost every Markdown implementation out there (Discord's excluded) already uses _ for emphasis. I don't think there's any other 2-characters mark that would work for underline. Anything longer is clunky and generally worse than using .

Yes but I don't remember it exactly. In the original Markdown they behaved identically but over time it became apparent that it was almost never the case that users meant to emphasis words that way. Most often, users would type some_random_words and didn't want the middle part to be emphasized. On the other hand, when they used * it was almost always to signify emphasis. You can find discussions about it by searching for "intra-word emphasis" and un*fucking*believable markdown.

~~That could be a bug. I'll look into it.~~ What's the expected output? Is that from another implementation? Edit: not a bug, it's just a case of undefined behaviour. The first two tildes are paired with the next two tildes so you end up with something like <s>m</s>m~.

Pollux

JoshyPHP I'll look into it this week. The superscript behaviour (especially a^bc) is mostly based off Reddit's behaviour at the time it was implemented. I might make the implementation aware of word boundaries based on ASCII punctuation but I want to think about it carefully. Many Markdown implementations base establish work boundaries off ASCII letters, which can cause issues with non-latin languages.

I mentioned my post from two years ago only because you asked for other similar discussions and that one I remembered. I'm absolutely fine with the behaviour that we have right now. Superscripts can be closed as expected.

matteocontrini Markdown does not have underlined text in general

JoshyPHP I wish that was the case as well but almost every Markdown implementation out there [...] already uses _ for emphasis.

95% of my users never heard about Markdown, another 5% have heard about it but don't care a jota whether it's implemented properly, unproperly or not at all (some other ruleset instead). I know that other communities are different, I just wanted to explain where I come from.

I would love to have the option of defining my own rules. As this doesn't seem like a realistic prospect, I will probably try to write my own markup extension forking the existing Markdown extension. I have no idea though how difficult this will be.

To explain my ideal solution in more detail: As my highest priority I would like to have bold, italic, underline, subscript and superscript with single delimiting characters as well as quotes, links and embedded images. Second highest priority would be ordered and unordered lists, probably tables. Third highest priority would be headlines, rulers and code, they are hardly ever used by anyone except me. And I use code tags only for troubleshooting regarding the current Markdown.

JoshyPHP Yes but I don't remember it exactly. In the original Markdown they behaved identically but over time it became apparent that it was almost never the case that users meant to emphasis words that way. Most often, users would type some_random_words and didn't want the middle part to be emphasized. On the other hand, when they used * it was almost always to signify emphasis.

I understand, so this problem would arise as well if the underscores would denote underlined text.

JoshyPHP ~~That could be a bug. I'll look into it.~~ What's the expected output? Is that from another implementation? Edit: not a bug, it's just a case of undefined behaviour. The first two tildes are paired with the next two tildes so you end up with something like <s>m</s>m~.

Well, the expected outcome would be to close tags in the reverse order of their opening: <s>m</s>m. But I'm aware that this would require regex lookbehinds, probably somewhat costly.

On the other side: In the case of bold and italics it works as expected:

***m**m* >> mm, ***m*m** >> mm

The first sequence becomes mm, the second mm.

JoshyPHP

Pollux I mentioned my post from two years ago only because you asked for other similar discussions and that one I remembered.

Oh ok. I was more thinking about discussions about other Markdown libraries, feature requests on other forum software, other commenting systems, that kind of thing.

Pollux I would love to have the option of defining my own rules.

Simple markup can be implemented via the Preg plugin. It's only alright for the simplest things. Otherwise, I'd recommend creating a new plugin rather than extending one.

$configurator->Preg->replace('/__\\B(.*?)\\B__/', '<u>$1</u>', 'U');

Alternatively, you can prepend a tag filter that intercepts STRONG tags and replaces them with your own:

$configurator->tags->add('U')->template = '<u><xsl:apply-templates/></u>';
$configurator->tags['STRONG']->filterChain->prepend('formatUnderline')
	->resetParameters()
	->addParameterByName('tag')
	->addParameterByName('parser')
	->addParameterByName('tagText');

function formatUnderline($tag, $parser, $tagText)
{
	if ($tagText === '__')
	{
		$tag->invalidate();
		$parser->addTagPair('U', $tag->getPos(), 2, $tag->getEndTag()->getPos(), 2);
	}
}

Pollux Well, the expected outcome would be to close tags in the reverse order of their opening

Ok, actually I couldn't figure out what the output was meant to be.

The algorithm that handles emphasis is stupidly more complex than the generic routine that does strike-through and inline spoilers. I prefer to keep it as simple as I can, especially if there's an alternative syntax that can be used, e.g. ~(~~m~~m)

JoshyPHP

I'm revisiting the underline snippet and I realize the tagText parameter was added in v2.0 of the library. In v1.x you'd need to get the text and manually capture the substring consumed by the tag. https://s9etextformatter.readthedocs.io/Filters/Callback_signature/

Pollux

JoshyPHP The algorithm that handles emphasis is stupidly more complex than the generic routine that does strike-through and inline spoilers.

Such a stupid complexity is to be expected when one character is used for different kinds of formatting.

I prefer to keep it as simple as I can, especially if there's an alternative syntax that can be used, e.g. _mm

I didn't even know that there was such a syntax, even more so I wouldn't expect my users to keep in mind such workarounds. I would therefore prefer to use different characters for different purposes, and those characters should be as easy as possible to remember: * for bold, _ for underline, ^ for superscript, ~ for subscript.

I just need to find a convincing character for italics. The slash looks like a natural candidate, but problems (inadvertently italicizing words) have to be expected, so we have to think further.

One might restrict single character delimiters to work only within words (including whole words) and delimiters of double characters of the same kind to work inside a paragraph (including whole paragraphs). Triple characters might work spanning more than one paragraph:

This is un*fucking*believable markdown. and This is a *bold* word.

This is un/fucking/believable markdown. and This is an /italic/ word.

**This is a bold sentence.** and These are //more than one words in italic.//

***These are two... ... bold paragraphs...*** ... and ///this here... ... is italic///

But I agree, this wouldn't be markdown anymore, it would just be my preferred choice of intuitive formatting.

jordanjay29

Pollux The slash looks like a natural candidate, but problems (inadvertently italicizing words) have to be expected, so we have to think further.

URLs come to mind, especially when dealing with relative stuff here like path/to/your/flarum.

Pollux

jordanjay29 URLs come to mind, especially when dealing with relative stuff here like path/to/your/flarum.

URLs should be handled before anything else, otherwise desasters have to be expected, not just with slashes. Think about URLs like https://en.wikipedia.org/wiki/United_States_of_America for instance.