Help with a >3 million users Flarum

JCKodel

I have a Flarum setup with > 3 million users and I'm having some issues regarding performance (kind of).
The forum responds very well, but sometimes it freezes for several seconds. It seems like it opens 2 or 3 pages really fast, then freezes for several seconds, then is fast again.

I'm seeking for information regarding how can I investigate the issue.

Flarum info: 0.1.0-beta.13, PHP 7.2.24-0, MySQL: 5.7.33.
Users: 3,239,811
Discussions: 8,390
Posts: 38,905

Server: Ubuntu 18.04.5 LTS (Bare metal)
RAM: 256Gb
HD: 256Gb
CPUs: 32

CPU is free (load average: 1.96 2.03 1.89)
Mem is 9.87G (from 256Gb available)

I don't have too much experience with MySQL (for MSSQL, I would run the SQL Profiler, wait some minutes and then it would tell me exactly which indexes are missing, etc.)

Server is running PHP via FPM on Nginx.

pm in on demand, max children = 256, start servers = 32, min spare servers = 16, max spare servers = 64, process idle timeout = 60s, max requests = 1024

clarkwinkelmann

JCKodel Flarum info: 0.1.0-beta.13

Have you tried Flarum beta 15 yet?

Extensions might also have improved their performance in the latest versions that are only compatible with the latest Flarum.

Justoverclock

This happen only on your pc or for everyone? Could u share the link?

JCKodel

Justoverclock I'm guessing for everyone. There is no way to know.

My PC is a Threadripper 3970x (32 cores, 256Gb RAM, nvme 5tb/s), hardware is not an issue (but I know there is a zillion more things to consider)

The link is https://mcc-forum.code.art.br

To see the behavior, open 15 pages in the background (I use the mouse middle button), then try to scroll (there is an autoload, so pagination is automatic).

After 1 or 2 pages, it freezes.

I don't know if it is an server issue, if it is a client issue (perhaps too many connections to the same host), or if it some PHP setting that sleeps after some time and the first run is problematic.

I usually open it in the morning and open in the background all unread threads (so I can moderate them). After all pages are open (or ordered to open), I check the Flag button (posts signaled for moderation). That menu will take several seconds to open.

But, most of the time, the forum is very performatic. Right now I'm trying to simulate the problem I have every morning and it seems to be very responsive. I'm really confused about what exactly is happening.

Everytime you show someone something that is broken, it auto magically fixes itself >.<

luceos

Perhaps it's because Flarum updates your "last read" pointer.. Other than that .. maybe enable slow query log?

JCKodel

luceos What this means? What is "last read" pointer and how I disable (and what implications disabling it have)?

Forgot to mention that admin panel always takes 1 minute or more to load.

How can I enable this slow query log?

@Ralkage I disabled the Discuss view plugin and cleared the cache in the admin page (not sure exactly what that does), no difference (the post with more views still loads slower than the other one). I'll keep this plugin disable until tomorrow to check if it does make any difference. Thanks for the info.

Ralkage

I did notice this slowness when loading discussions that have a ton of views.

https://mcc-forum.code.art.br/d/7-hidratacao - this discussion has 2M views and just one post where as this discussion has 6K views and very few posts. Can you try disabling the Discussion Views extension and check if the 2M views discussion takes a long time to load still?

rob006

Ralkage I did notice this slowness when loading discussions that have a ton of views.

This is rather related to number of reactions. In HTML source you can find that there is a huge JSON with data of all users that reacted to this post. So I would rather start with disabling reactions plugin.

Ralkage

rob006 could possibly be the case; @JCKodel can you try enabling Discussion Views again and check again?

JCKodel

rob006 I disabled FoF Reactions (and cleared the cache), but no change.

The page itself takes 7.48s to load (and is only 124Kb). There are no requests with huge amount of data (50, 60kb, including images and scripts).

Not sure why it takes 5 to 8 seconds to load in the first place. And why it always loads (it seems the first request is never cached)

JCKodel

Ralkage Enabling or disabling Discussion Views or FoF Reactions doesn't seem to change anything.

With Discussion Views enabled, the page takes 5 to 7 seconds to load (then it triggers all other loads). There is no request larger than 20k (excepting some JS, images, etc.).

The home index (discussions) is 85Kb (with Discussions view turned on)

Opening a page with 100k views takes 587ms. That page with 2 million views takes 5.26 seconds.

With Discussion Views off, the page is slower. A Ctrl+F5 took 17.8 seconds, a simple F5 takes 15.4 seconds!!!! (not sure if is the plugin or if the original slowness problem is hitting me)

Enabling now Discussions View and FoF reactions and clearing the cache...

The page with 2 million views took 26.56s to load (after one initial load to desconsider the cache clean up).
A second F5 and it takes 15.95s. All requests, except the initial request, a 580fxwdgus script are cached (disk or memory cached by MS Edge).

@rob006 With FoF Reactions on and a Ctrl+F5 to disregard browser cache, still there is no huges requests. There is a base.js with 516kb (but it takes only 39ms to load), followed by the page itself with 164Kb (and 6.69s to load).

I don't think FoF Reactions is being used or make any difference (I'll leave it disabled, nevertheless)

rob006

JCKodel I disabled FoF Reactions (and cleared the cache), but no change.

It definitely reduced size of this JSON. I would try also disable webpush notification plugin and likes, and experiment with these tree plugins to check which one has the biggest impact on performance.

JCKodel

rob006 Found our 1Mb JSON. But it was from FoF Terms. Disabling this plugin makes that page load in 2s. Waaaaay better than 15, 20...

Opening some 10, 15 pages, the initial load is 500ms to 1.5s. I guess that was it. Not perfect, but at least I think it will be faster now.

Thank you all for the help.

Just one final question: it is normal for the initial request not to be cached by the browser?

JCKodel

clarkwinkelmann I have some traumas playing with Flarum on command line (everything stops working, cache problems and error messages everywhere). I cannot afford to upgrade now.

rob006

JCKodel For me https://mcc-forum.code.art.br/d/7-hidratacao takes over 700 kB and contains metadata of all 2k users that liked first post (even if most of them ere never visible on that page). Before disabling reactions plugin, it was 2x heavier. I'm pretty sure that fetching and serializing info about 2k users have noticeable impact of server resources and response time.

Also note that many of usernames on your forum are generated from email, and since username is publicly available in API response, you're leaking emails of these users, if they have some activity on your forum.

JCKodel

rob006 Yes, I know. Problem is there is no user registration on this forum (my users come from an Android/IOS app and they are registered first time they use the forum, using their app user). Since Flarum doesn't support user names (with spaces, etc.), I wrote a plugin to use bio info as the user name (and the flarum original user name is the email itself, so the user search by name (mention) works... most of time =P)

Ralkage

JCKodel well, when you are eventually able to upgrade to the latest version of Flarum, we have an official Nicknames extension now 😉

JCKodel

Ralkage That's tempting.

It would be awesome instructions on that page in how to create nicknames through API (because I use SSO)

askvortsov

With the nicknames extension enabled, all it'd be is sending a request to the edit user endpoint, with the new nickname in data attributes as per the json API spec.

Justoverclock

your forum is slow, for sure, but i can't reproduce "freeze"....i have high end pc also and a gigabit connection.