Poor performance after server migration.

Shock

The issue

We have a very simple installation with minimal extensions installed, and a very small community of about 50 people posting in one discussion per day that reaches around 500 replies. We are experiencing some consistently poor performance when opening and scrolling through discussions and I would like some help to try to diagnose the issue.

We were previously on a shared hosting environment and actually did not have these problems (but had other problems,) and we recently moved to a VPS where we are consistently seeing 4-6 second load times when loading a discussion.

A few things I've noticed:

1) Resources: No significant resources are being used, only 20% memory and <10% CPU is ever in use.
2) Does not seem related to size of discussion. All other topics I've found seem to talk about poor performance when discussion is large. In our case we get poor performance no matter how many replies there are to the discussion. Even a discussion with 20 replies takes 5 seconds to load.
3) Does not seem related to load. Performance is the same if many users are active or if site is dead.
4) MySQL settings seem to all be correct. It has 4GB of RAM allocated but is only using < 2GB.
5) I enabled slow_query_log and tried to find any problematic queries but really doesn't seem to be database-related. Moreover, the database is the same as it was when we were on the shared hosting environment so it's unlikely to be a case of missing indexes or anything like that.

Just looking at the API Calls, for example calling /public/api/posts regularly takes around 4 seconds. Compared to here on discuss.flarum.org the same API call returns in < 1 second.

I really cannot figure out what is going on so I am asking for some new ideas.

Thank you in advance.

Flarum information

Flarum core: 1.8.3
PHP version: 8.1.25
MySQL version: 8.0.35
Loaded extensions: Core, date, libxml, openssl, pcre, zlib, filter, hash, json, pcntl, readline, Reflection, SPL, session, standard, bcmath, calendar, ctype, curl, dom, fileinfo, ftp, gd, iconv, imap, mbstring, mysqlnd, PDO, Phar, posix, SimpleXML, sockets, sqlite3, tokenizer, xml, xmlwriter, xsl, zip, mysqli, pdo_mysql, pdo_sqlite, xmlreader, xdebug
+------------------------------------+----------+------------------------------------------+
| Flarum Extensions | | |
+------------------------------------+----------+------------------------------------------+
| ID | Version | Commit |
+------------------------------------+----------+------------------------------------------+
| flarum-suspend | v1.8.1 | |
| flarum-tags | v1.8.0 | |
| the-turk-quiet-edits | 0.1.3 | |
| fof-user-directory | 1.3.2 | |
| fof-user-bio | 1.3.0 | |
| fof-recaptcha | 1.3.0 | |
| fof-nightmode | 1.5.3 | |
| fof-ignore-users | 1.2.0 | |
| fof-forum-statistics-widget | 1.1.0 | |
| flarum-subscriptions | v1.8.0 | |
| flarum-sticky | v1.8.0 | |
| flarum-statistics | v1.8.0 | |
| flarum-nicknames | v1.8.0 | |
| flarum-mentions | v1.8.2 | |
| flarum-markdown | v1.8.0 | |
| flarum-lock | v1.8.0 | |
| flarum-lang-english | v1.8.0 | |
| flarum-bbcode | v1.8.0 | |
| clarkwinkelmann-predefined-avatars | 1.0.1 | |
+------------------------------------+----------+------------------------------------------+

luceos

Shock you left out the bottom section of the info output. It shows whether debug is on. I can imagine that could be one of the causes, we need to confirm.

Shock

luceos Sorry, I didnt mean to leave that out. debug_mode is certainly turned off.

RelicSystem

Curious, what web server are you using and how are you using PHP - mod_php (apache) fpm, fastcgi etc. I tend to lean towards nginx myself with php-fpm 8.2 and mariadb/percona over standard MySQL..

My first point of call would be to check (again in my own setup) that the websites php cool is configured properly and has enough workers assigned on both the web server end and php-fpm end.

Wulfheart

Wild guess: Has opcache been previously enabled and is now disabled?

Shock

Wulfheart We originally did have it disabled. We have enabled it now. We are running Apache and PHP-FPM which has 25 workers.

We've managed to do enough that our call of the /posts endpoint is noticeably faster, but it is still consistently 1-2 seconds compared to <1 second here on flarum.org. We are running out of ideas of what else we can look into. I do not see any bottlenecks at the database level or in terms of any resources, and for PHP-FPM I only see a dozen or so processes active so I am confident it has enough workers.

Moreover, it doesn't really seem to matter what the load is on the site. If it's 4 AM and I am the only one using it, the performance is the same as it is during peak usage. Also doesn't seem to matter what extensions are enabled and we use very few extensions anyway.

GreXXL

Shock can you share a URL to your site to experience the issue firsthand? Maybe you can explain your setup a bit more detailed also.

Shock

GreXXL Unfortunately I cannot provide a URL as it is not public. I am not sure what more I can provide for the setup. It is a VPS with Apache/MySQL. PHP-FPM enabled with 25 workers, OpCache enabled. Everything is the latest version. Compression is enabled.

If you request an endpoint for example here on flarum.org by going to posts/filter you can see that on flarum.org it returns in about 400-500ms, however doing the same thing on our private website takes 3-4x as long, even though the size is actually smaller (4.8 KB compared to 5.5 KB in that flarum.org example.)

As mentioned, it also does not matter if the server is experiencing high or low load.

I am really a bit stumped as to what the bottleneck could be.

luceos

Shock have you tried disabling all 3rd party extensions to see if that helps?

GreXXL

Shock If it's running internally, maybe you are embedding or requiring some external content (recaptcha?) that is blocking and reducing loading times. Other than the server, what did change between migrations? Maybe its some sort of MySQL configuration issue or some indexes that need to be rebuilt?

rob006

Is MySQL on the same server as PHP?

dooooove

I've encountered this issue too, and it's quite frustrating.

GreXXL

dooooove you will have to share a lot more information then that to be to help you. (https://docs.flarum.org/troubleshoot)