CyberGene we've been playing with https://www.stackhero.io/en/services/Elasticsearch/pricing
What’s the cheapest solution for a better search?
The Scout extension adds compatibility with Algolia and Meilisearch. Algolia has a free tier and Meilisearch can be self-hosted. It's still a new extension so there might be some issues but I will fix them rapidly should any arise. I have not heard any feedback from someone trying it in production yet, but I'm planning to use that extension with Meilisearch in production myself soon
clarkwinkelmann Ohh, that's really nice, I will try it right now. I guess I can leave the sync driver for now? To my understanding, the added overhead would be that every post will be uploaded in the same thread as the posting one but since it's text search, it shouldn't take long, right? In case I notice any perceptible slowdowns, I will switch to a different queue driver, e.g. the database one. I will share my opinion in the other discussion.
CyberGene that's correct, it will cause a small performance drop when creating or editing a post, or when editing a user profile.
If you're using one of the externally hosted indexes like Algolia I would recommend a queue. Any queue driver would be fine, as it generally doesn't matter how long it takes to index as long as it happens in the next few hours. This way your users won't be impacted if Algolia has some degraded performance and takes longer to complete requests.
Just like with emails, using an async queue also has the benefit of hiding any error from the user. They might be confused to see an error message after saving their posts and try to create the post again, even though the post was successfully created and the error just happened afterwards while events are being handled by Flarum.
Teddan Thanks for suggesting sonic, however I'm on a shared hosting and can't deploy anything there, I have to use on an external service. I use AWS though but I'm not sure Sonic is supported there or would be cheap if possible? I'll have to check though.
Teddan sonic
however doesn't support ranking. In my experience using a search engine without tuning relevance doesn't yield better results than Flarum's native search... I experimented with Elasticsearch and I came up with a 100-line query that also takes into consideration field boosting, likes, views, reduced impact of post length normalization (plus ASCII folding, decimal numbers separator normalization, synonyms etc.)
In my experience this kind of personalization is required otherwise results aren't satisfactory, but YMMV.
Hari Was it good? How much RAM it's occupying?
Meilisearch's storage is on disk and it loads data into memory only when querying. If you have little memory it will still work but it will be slower. Their recommendation is to have enough RAM to have the whole database loaded in memory.
matteocontrini i have almost 30k discussions and it will grow to 70k discussions in the next 10 months. we are using our flarum as a backend application to address user needs, most users will not come to my flarum and perform actions only the admins or mods use it to find similar threads and try to post existing answers so i do not think there will be much user search queries but i am sure this will change after one year. (i will give an interview on this to gXXL after few months, it is a unique case study how we are using flarum)
like mentioned here Hari i am looking for full-sentence matches in post content
search
i have 4GB RAM with 2vCPU Plesk setup running for only flarum and 2-3 WordPress subdomains.
do you recommend meilisearch for my need? since you have explored all the possible ways you are the right person to get suggestions from.
- Edited
Hari do you recommend meilisearch for my need?
I have a similar volume and when I tried Meilisearch it worked quite well. Queries were fast (few milliseconds) even on the 600k posts collection. I'd say give it a try, I think it's the best trade-off between cost and performance/features/quality at the moment. The only issue I've found is that indexing can be quite slow (much much slower than Elasticsearch) but I know they're working on it.
I personally didn't go for it because I found out that Elasticsearch is not that expensive in terms of resources as I anticipated, and it's much more customizable. Also, it provides an API to update documents in batch by query, something that is useful when you need to keep the search database in sync with MySQL.
Hari i am looking for full-sentence matches in post content search
Do you mean exact match of a sequence of words? Meilisearch by default takes into consideration proximity for ranking search results, you can tune that: https://docs.meilisearch.com/learn/core_concepts/relevancy.html
It also supports using double quotes "
to force and exact query: https://docs.meilisearch.com/learn/what_is_meilisearch/features.html#phrase-search
Sonic doesn't support phrase queries for example, while in Elasticsearch you would do that with the match_phrase
query.