True, AJAX is not the problem here.
Hmm, I did a bit more digging (quite time constrained at the moment) and a bit piece of the puzzle seems to be latency between my computer and the Flarum server. That's approx 150ms which adds up.
According to a quick test I'm first waiting 600ms to receive the initial page. Then an additional 600ms is needed for forum-en-*.js to arrive. That makes 1200 ms before anything meaningful can happen.
But I guess I should benchmark with a server with a bit less latency to get a better picture of performance.
A quick fix for Flarum.org would be to either use HTTP/2 to push the CSS/JS resources or use a global CDN for static files. That would eliminate almost 600ms of a (first) pageview.