Wish to extend the limit per post to a million characters instead of 65,536

Robert_Walker

We need to have long posts in our new forum occasionally. I hit the limit for our rules but it would be great to have longer posts in other contexts too.

Following it up I find that it is an artificial limit in Flarum that goes back to 2020 when the database had a maximum of 65,535 characters per row in TEXT format which lets you store the data a little more efficiently, but it is only a saving of a few bytes per row.

However checking the database for our forum, Flarum has changed the database content type to permit millions of bytes per row. So, there is no longer any reason to have this limit. It is just a historical thing. Low priority presumably.

I've found out that MySql databases can usually store over a million multi-byte characters per row, enough for an entire book even when set to MEDIUMTEXT.

The Flarum posts column is indeed set to MEDIUMTEXT.

[upl-image-preview uuid=356e62b2-da0b-4297-b6b4-7a790ef11d9c url= alt={TEXT?}]

TINYTEXT: 255 characters - 255 B
TEXT: 65,535 characters - 64 KB
MEDIUMTEXT: 16,777,215 - 16 MB
LONGTEXT: 4,294,967,295 characters - 4 GB

https://www.atlassian.com/data/databases/understanding-strorage-sizes-for-mysql-text-data-types

When someone else asked the question they got the answer:

QUOTE MEDIUMTEXT should already be able to store over 16 millions characters, so this 65535 limit does not come from database.

https://discuss.flarum.org/d/25309-increase-the-content-character-limit

I also found that the origin of the limit of 65536 goes back to 2020 when the column was TEXT.

QUOTE STARTS
This is a limitation of the MySQL text field that we use to store content. While this could in theory be adjusted to a MEDIUMTEXT or LONGTEXT field it requires more store space (2KB overhead for TEXT, 3KB for Medium Text, 4KB for LONGTEXT) which adds up a lot with large numbers of post. Further it's incredibly rare for people to post that much data. For that reason it's not the default.
https://discuss.flarum.org/d/25007-the-content-may-not-be-greater-than-65535-characters/2

From the screenshot I just shared it's clear that the My SQL field is now set to MEDIUMTEXT.

So it should be possible to extend it without adjusting the database if I understand right.

Perplexity AI suggests I edit a file called extend.php in the Flarum folder and gave me this code:

<?php
use Flarum\Extend;
return [
    (new Extend\Validator(\Flarum\Post\PostValidator::class))
        ->configure(function ($flarumValidator, $validator) {
           $rules = $validator->getRules();   
            if (!array_key_exists('content', $rules)) {
                return;
           }    
            $rules['content'] = array_map(function (string $rule) {
                if (\Illuminate\Support\Str::startsWith($rule, 'max:')) {
                    return 'max:1000000';  // Set to 1 million characters
                }
                return $rule;
           }, $rules['content']); 
            $validator->setRules($rules);
        }),
];

[edited to remove the back ticks see replies]

Chatbot discussion here:

https://www.perplexity.ai/search/is-there-an-extension-or-setti-1wVHM_KwTieeAxFHztGz2Q Is there an extension or setting to increase maximum length of Flarum post beyond 65535 characters

It looks simple enough. But there is no way that I will just use code written by a chatbot without knowing how it works, not for something like this.

So that's my question. Is it as simple as that and is this code okay to use or does it need to be done slightly differently, or is there another way to do it?

I'm a coder myself, a developer of microtonal and polyrhythmic software for Windows, and I can probably add some extensions myself at a later date but I'm new to all this right now.

luceos

Robert_Walker i didn't fully check the code, but aside from the back ticks it does look great. Did you try it?

clarkwinkelmann

Not sure how many other people used the same syntax, either independently or modifying from an example, but I'm pretty sure this strategy of using an array_map to modify validation rules is something I first used in extensions and local extenders. The AI must have learnt from the bests 🤣

Same comment as @luceos , the extender looks like the right approach and the code seems good, but I haven't tested it.

Something to note, if you're testing with data close to the character limit, the database column size and the validator are for the XML parsed version of the post content, which is always longer than the input text. Depending which formatting features are used in the post, the XML version can be much longer than the original text.

As for the default limit, I'm not sure how we ended up with a larger database column without changing the validation, but I feel like the default limit is somewhat reasonable, and it's not too difficult to modify via extend.php. I think it's best to have a lower limit that can be raised, rather than an incredibly high default value that some forum owners might only notice when someone does create a way too long post.

Robert_Walker

clarkwinkelmann

Great that worked 🙂

https://robertinventor.com/Flarum/public/d/79-test-of-a-long-post-and-replies-alice-in-wonderland-wind-in-the-willows-wizard-of-oz#undefined

Yes Perplexity AI learnt from you, it gave you as a cite for its code

First it suggested I edit the validator directly, and it warned me that every update would overwrite it.

I asked it to check for discussions of how to update it, and then it wrote the code I shared here.

I then asked it for human sources and it didn't find the exact code but shared its own extend.php code again saying it was based on your examples here:

https://discuss.flarum.org/d/29506-custom-user-validation-rules-examples

Yes on the XML - our rules were only 56,777 characters but got the warning about 65,536 characters, so I expect it's the extra XML that took it over.

After this update to a limit of a million characters, I've tested it with half a million characters, using ten copies of the rules and that also worked and tested it with three medium sized books, Alice in Wonderland, Wind in the Willows and Wizard of Oz.

Also tested it with one of my own fact checks that our group often uses with 200,000 characters.

All fine in both posts and comments. And including many images in my own 200,000 character fact check.

One more question. Though nobody is likely to reach the 1 million character limit can I safely set it to the maximum for MEDIUMTEXT of 16,777,215?

Just no need in our particular group for any limits on the text at all, except what's required by the My SQL limit.

clarkwinkelmann

Robert_Walker yes, there shouldn't be any issue allowing the maximum length.

The only potential problems I can think of is the server storage filling up quicker and the search Index getting slower if many people take advantage of the full length. But the same will also happen with a lot of short posts.

Robert_Walker

Okay great 🙂. I don't expect many people to take advantage of it. It's just not to have any artificial limits. After setting it to a million, why not just set it to the maximum for MEDIUMTEXT, so I'm glad to hear that's okay to do and I've done it.

Highly unlikely to go far above 200,000 in reality.

After making that change, I've just tested it with War and Peace, at 3.3. million characters, uploaded no problem though with a few seconds pause after hitting reply to update it.

I don't expect anyone to write a fact check post or comment that long 🙂.

In principle though, it suggests that Flarum could be used even by e.g. authors who want to share entire books they are working on in a single post, to discuss with other authors.

Though we are not likely to do posts this long, going beyond the limit of 65,535 is a major plus for us, and indeed having topics and comment replies that can be very long with embedded video and graphics and rich text is a very big plus of Flarum for us.

We are voluntary fact checkers for scared people and on some topics my fact checks are very detailed and run for many pages.

Now I can do the equivalent of complete Substack posts as a reply when needed - and it means that other debunkers who don't have blogs / Substacks can also reply in similar detail without the artificial restrictions we had before as a Facebook group.

clarkwinkelmann

Very long posts can lead to other issues outside of the storage limits:

Parsing, un-parsing and rendering takes longer
The parser memory requirements could go above the PHP limit, depending which extensions are active and which formatting features are used in the post
The discussion page and scrubber might be difficult to navigate if a single post occupies many times the viewport height

Robert_Walker

Okay. I've not noticed anything yet.

Main thing I wondered, I've got this test post with several million characters in the original post and the replies.

Would a single post very long post just for testing, that nobody is going to be much interested in cause any problems. It's not that I expect anyone to come to our forum to read War and Peace 🙂.

It's just to demo what it can do. I can delete it if it is likely to cause problems in the future.

If interested in a bug report, I did spot a bug with the table of contents for the very long post that the chapter links for the table of contents extension often didn't work and I assumed it is some kind of overflow problem of some sort.

But not one that needs to be fixed 🙂. Given that it is way over any likely limit in practice.

That's all I noticed - even War and Peace didn't seem to cause many problems.

Generally pretty impressed by its ability to handle so much text.

Robert_Walker

Update: may be of interest to any others who want to increase the length of their posts similarly. I got a report from a user that our forum was getting a bit slow and on testing the search took 4 seconds to show up - and after removing that test long post and replies and a couple of other long posts it was down to 1.5 seconds.

I had stress tested it by adding over 4 million characters in a test post and comments, content that our forum didn't need just to test the limits.

So I can confirm that adding:

https://www.gutenberg.org/files/11/11-h/11-h.htm Alice’s Adventures in Wonderland 152,735 chars

https://www.gutenberg.org/cache/epub/289/pg289-images.html The Wind in the Willows: 353,040

https://www.gutenberg.org/cache/epub/55/pg55-images.html The Wonderful Wizard of Oz 234,056

https://www.gutenberg.org/cache/epub/708/pg708-images.html THE PRINCESS AND THE GOBLIN 295,841

https://www.gutenberg.org/cache/epub/709/pg709-images.html The Princess and Curdie 319,820
https://www.gutenberg.org/cache/epub/2600/pg2600-images.html War and peace 2,490,239
and a post of my own of 202,213 characters was enough to slow down the search noticeably though only by a couple of seconds.

That's 152,735 + 353,040 + 234,056 + 295,841 + 319,820+ 2,490,239 + 202,213 = 4,047,944

So - although it's possible to have very long posts, I can form that when you get to many millions of characters it does slow down the search a bit and it might have the other effects you describe which I didn't notice.

So I've removed the posts. And advised members that we encourage long posts and replies but it's not the best place to compose a novel rivaling war and peace in length 🙂.

Not likely to be a problem, I'm the only member who has the time to write such detailed fact checks at present and I have my own substack for any very long posts.