Hello everyone 👋 ,
Something is brewing in the "working from home" offices of the Blomstra team. An upcoming client is having a bitter experience with the native Flarum MySQL search functionality. Having tens of millions of posts means that enabling the search bar on their community causes heavy spikes on their extremely heavy virtual machine. Their intent to become a client as soon as possible made me consider the impact on our managed hosting platform and their perception of our service. It was important to me to prevent any disappointment! Especially when they state "In Luceos I Trust", I kid you not 🥰
For months I had this idea in my mind how I would solve search for enterprise communities would I have the time to dive into it. No reason for delay anymore it seems! So I sat down for hours on end on Thursday; I saw the sun go up, take it's journey along the sky, dip into the far reaches of the horizon. Just after midnight I had a first version running, a surprising success! Search that relies on a database meant for fulltext searches. A drop-in replacement for the native search even.
This first concept was built on top of Meili search, a funky little search database. Quite easy to install anywhere, to seed and to search with. A test with only 50k documents (discussions) stored into Meili took a search 4759ms (almost 5 seconds) to run using its native administration panel. That wasn't acceptable.
A quick talk in the team later we settled on managed elastic search, because we want to service our customers with something reliable, performant scalable and proven. A work day later the extension was refactored to seed posts into a very small managed elastic instance. Although only tests on a larger database makes sense it currently:
- responds between 200ms and 400ms (50k posts)
- understands tag permissions
- understands byobu permissions
- allows searching for full sentences and complex combinations
- allows searching posts and discussion titles
- does not yet understand gambits
- supports Flarum native sorting rules
So why post about this? Well, I'm excited and I'm even more excited to announce that this extension will be 100% open source and free to use in the near future! Just see it as a gift from team Blomstra to the Flarum ecosystem. We're hoping to see this extension adopted by larger communities and willingness to collaborate on making it even better.
Only after having started work on this elastic integration did I discover the Sonic extension. Although a very interesting extension to have, I think for the largest of our clients would need a managed service to rely on to deliver consistency and quality.
Roadmap
Before a beta release
- improve relevancy values
- test the implementation on a multi million posts community
- consider user search (do we need that?)
Before a stable release
- add extenders
- get a review from the core team
FAQ
So when can you get your hands on this?
I first want to test this extension on the acceptance environment for our future client. Testing against a community with a huge amount of posts can validate that this indeed is the right solution for us, but also for communities feeling those growth pains.
Do I need this?
Not everyone needs this. Only once you notice searches take longer than 500ms on your community it's time to consider this extension, but only after you looked at your hosting capacity first. Usually only larger communities (going over a million posts) have an immediate need to delegate search away from their MySQL database (cluster) and webserver node(s).
