Also, remember Flarum is a SPA as a result it will load these static assets only when entering the site, not while browsing through it.
I think Flarum is fast already, there is still lots of improvement possible though, but most of that (from my backenders point of view) is related to how the database schema is set up and the lack of intelligent caching.