What’s the cheapest solution for a better search?

CyberGene

There’s some scattered information but I’m getting lost. Let’s ask this straight as of June 2022.

Is there a way to improve the search functionality in Flarum (like partial matches not just exact entire word, also search within posts, etc)? And if so, what’s the cheapest way to do so for a shared hosting Flarum, like which extensions and external (cloud) services to use?

I looked at ElasticSearch service but it’s ridiculously expensive, like $90 a month. I will be OK with $2-3 a month. It’s a small forum with 150 members and not very active, 10-15 posts a day.

GreXXL

CyberGene in case you don't know it - here is a link to some search tips that are very useful to know. What you mentioned is the Elasticsearch extension which improves search of discussions (especially in large communities). There are also shared hosting providers that allow using Elastic. As an alternative you could rent a own VPS / Droplet that are available for reasonable costs.

CyberGene

GreXXL thank you for the links. The first one with the tips is not really what I'm looking for. What my users miss is more free text search functionality, where you can start typing something to search for and you get matches even for the partially typed term, or match against the content, not just discussion titles, etc.

I think once I tried the Elasticsearch extension with the trial period of ElasticSearch own service but things didn't work due to a Flarum bug with wrong permalinks since I'm in a sub-folder Flarum installation.

I'm on Siteground and they don't provide ElasticSearch. What's the cheapest option for using an external ElasticSearch instance? I see there's one on AWS (I already use S3 for file storage from FoF Upload) but for the life of me I can't figure out how much it would cost...

luceos

CyberGene we've been playing with https://www.stackhero.io/en/services/Elasticsearch/pricing

clarkwinkelmann

The Scout extension adds compatibility with Algolia and Meilisearch. Algolia has a free tier and Meilisearch can be self-hosted. It's still a new extension so there might be some issues but I will fix them rapidly should any arise. I have not heard any feedback from someone trying it in production yet, but I'm planning to use that extension with Meilisearch in production myself soon

CyberGene

clarkwinkelmann Ohh, that's really nice, I will try it right now. I guess I can leave the sync driver for now? To my understanding, the added overhead would be that every post will be uploaded in the same thread as the posting one but since it's text search, it shouldn't take long, right? In case I notice any perceptible slowdowns, I will switch to a different queue driver, e.g. the database one. I will share my opinion in the other discussion.

clarkwinkelmann

CyberGene that's correct, it will cause a small performance drop when creating or editing a post, or when editing a user profile.

If you're using one of the externally hosted indexes like Algolia I would recommend a queue. Any queue driver would be fine, as it generally doesn't matter how long it takes to index as long as it happens in the next few hours. This way your users won't be impacted if Algolia has some degraded performance and takes longer to complete requests.

Just like with emails, using an async queue also has the benefit of hiding any error from the user. They might be confused to see an error message after saving their posts and try to create the post again, even though the post was successfully created and the error just happened afterwards while events are being handled by Flarum.

Teddan

Try sonic

Es is eating a lot Ram. I failed to have Meilisearch working ( index error don’t know the reason)

Agolia is expensive as I know.

Sonic is light and very easy to deploy. Though it’s not supporting to search with single character .

CyberGene

Teddan Thanks for suggesting sonic, however I'm on a shared hosting and can't deploy anything there, I have to use on an external service. I use AWS though but I'm not sure Sonic is supported there or would be cheap if possible? I'll have to check though.

matteocontrini

Teddan sonic however doesn't support ranking. In my experience using a search engine without tuning relevance doesn't yield better results than Flarum's native search... I experimented with Elasticsearch and I came up with a 100-line query that also takes into consideration field boosting, likes, views, reduced impact of post length normalization (plus ASCII folding, decimal numbers separator normalization, synonyms etc.)

In my experience this kind of personalization is required otherwise results aren't satisfactory, but YMMV.

Hari Was it good? How much RAM it's occupying?

Meilisearch's storage is on disk and it loads data into memory only when querying. If you have little memory it will still work but it will be slower. Their recommendation is to have enough RAM to have the whole database loaded in memory.

Hari

CyberGene try Meilisearch digital ocean for $5/month shared the link on the other discussion.

Also let me know your experience 😜

Hari

CyberGene let's discuss here about Meilisearch

Was it good? How much RAM it's occupying? I am thinking to run in a docker on my 4gb RAM droplet

What are the installation steps?

Hari

matteocontrini i have almost 30k discussions and it will grow to 70k discussions in the next 10 months. we are using our flarum as a backend application to address user needs, most users will not come to my flarum and perform actions only the admins or mods use it to find similar threads and try to post existing answers so i do not think there will be much user search queries but i am sure this will change after one year. (i will give an interview on this to gXXL after few months, it is a unique case study how we are using flarum)

like mentioned here Hari i am looking for full-sentence matches in post content search

i have 4GB RAM with 2vCPU Plesk setup running for only flarum and 2-3 WordPress subdomains.

do you recommend meilisearch for my need? since you have explored all the possible ways you are the right person to get suggestions from. 🙂

matteocontrini

Hari do you recommend meilisearch for my need?

I have a similar volume and when I tried Meilisearch it worked quite well. Queries were fast (few milliseconds) even on the 600k posts collection. I'd say give it a try, I think it's the best trade-off between cost and performance/features/quality at the moment. The only issue I've found is that indexing can be quite slow (much much slower than Elasticsearch) but I know they're working on it.

I personally didn't go for it because I found out that Elasticsearch is not that expensive in terms of resources as I anticipated, and it's much more customizable. Also, it provides an API to update documents in batch by query, something that is useful when you need to keep the search database in sync with MySQL.

Hari i am looking for full-sentence matches in post content search

Do you mean exact match of a sequence of words? Meilisearch by default takes into consideration proximity for ranking search results, you can tune that: https://docs.meilisearch.com/learn/core_concepts/relevancy.html

It also supports using double quotes " to force and exact query: https://docs.meilisearch.com/learn/what_is_meilisearch/features.html#phrase-search

Sonic doesn't support phrase queries for example, while in Elasticsearch you would do that with the match_phrase query.