Scout Search

clarkwinkelmann

Version 0.1.2

Fixed support for using Flarum with a database prefix
Fixed posts not syncing to index when created through a discussion or a reply
Fixed old content being synced when a post is edited

I recommend running the scout:import-all command to fix all existing broken records.

This update changes the way post content is indexed. Previously the markdown/bbcode version of the post from edit mode would be sent to the index. Now it's the rendered HTML version except it's stripped of any HTML. This should prevent hidden information from being leaked via search (example: old username if a mentioned user changed their username, content hidden with the hide bbcode/reply/likes extensions). But it also means image and link URLs are now no longer indexed. I will try to add an option to get those URLs indexed again in a next version.

CyberGene I was not able to replicate your issue on Meilisearch, however I found out the synchronization wasn't working correctly. So maybe your new post wasn't indexed yet.

If the problem happens again, can you try performing the search directly in Algolia's dashboard? To see if any result exists in the search index? Meilisearch has a nice built-in web interface that allows inspecting the current state of the index and I believe Algolia has something similar.

CyberGene

clarkwinkelmann thanks, I installed the update and reimported the index. Seems like the search issue I had is fixed now, so maybe the problem was with what you described as possible reason. I will let you know if there are some other issues. So far all looks pretty good, thanks a lot! 🤝

Hari

They got opensource version too 🧐 I will try this

https://docs.meilisearch.com/learn/cookbooks/digitalocean_droplet.html

CyberGene Algolia is expensive, go through this old discussion luceos

CyberGene

Hari Thanks, I may check that later after I see how much Algolia will cost me this month.

CyberGene

Hari what's this digital ocean service, is it verified? I'm trying to register myself and can't go through the initial payment validation. I tried every possible payment method: Apple Pay, PayPal, direct credit card with all my cards and nothing goes through, they are all rejected for some reason. I'm giving up...

Hari

CyberGene as far i know you need not to pay any money to set-up a $5 droplet 🤔 they do accept debit cards too. And you get $100 free credit as a sign-up bonus (not sure about this as they advertise this)

CyberGene

Never mind, registered with Revolut, maybe my bank is having some problems currently.

CyberGene

clarkwinkelmann If you are regularly running composer update on all your dependencies, you should add an explicit requirement for the Meilisearch SDK in your composer.json because the extension requires * which might jump to a newer Meilisearch SDK version as soon as it comes out.

Can you tell me what exact stuff should I put in the composer.json?

I switched to Meilisearch in DigitalOcean, as explained by the link Hari posted. I imported the indexes, all works. I guess I will use it instead of Algolia because it's cheaper. Thanks Hari

Hari

CyberGene let's discuss here about Meilisearch

Was it good? How much RAM it's occupying? I am thinking to run in a docker on my 4gb RAM droplet

What are the installation steps?

tohsakarat

Hello, I use the extension with meili search and it works well, thank you!
However I find that the results that match content is prior to those match title. How shall I change the priority?
Thank you!

clarkwinkelmann

tohsakarat unfortunately it's a consequence of the current implementation, so can't be changed.

I'm not sure what the best way to calculate the sort order is. The problem is there are 2 indexes, one for discussion data (title) and one for post data (content). And I can't get the Meilisearch matching score out of Scout, only the order of the results. So merging the 2 in a way that makes sense is quite challenging. I guess it will require some specific code to go deeper level into Meilisearch API or modify the logic to use a single index.

clarkwinkelmann

Version 0.2.0 - TNTSearch

Following a suggestion by @010101 I have added TNTSearch support to the extension.

Existing users don't need to install this update if they are not planning to try TNTSearch.

To update:

composer require clarkwinkelmann/flarum-ext-scout

Additionally, to use TNTSearch you need to install an additional package. This package requires the sqlite PHP extension on the server!

composer require teamtnt/laravel-scout-tntsearch-driver

See the README on GitHub for more information https://github.com/clarkwinkelmann/flarum-ext-scout#tntsearch

If you switch driver, you need to rebuild the indexes with the command!

From my tests, TNTSearch seems an incredibly poor search engine. Their documentation is very lacking and the few options/syntaxes that are described don't seem to work. Maybe it can be improved slightly by playing with the settings, but I'm not sure it brings much improvement over Flarum native search.

This extension is still very experimental and designed to test alternate search engines with the goal of maybe integrating some of them natively in Flarum in the future.

fakruzaruret

clarkwinkelmann Why you prefered the TNT instead of Meilisearch?

010101

clarkwinkelmann That’s disappointing that TNT search ended up not being all that great. Maybe the devs for that will continue to improve it.

clarkwinkelmann

fakruzaruret I'm not sure what you mean.

The 3 drivers I've tested so far are Flarum native search, Meilisearch and TNTSearch. I have still not tested Algolia myself despite the extension supporting it.

I didn't see much of a difference between Flarum native and TNTSearch. I tried TNTSearch negative keyword syntax and tried searching sentences but it doesn't seem to work. It just works with simple keywords, much like Flarum native search.

Meilisearch is still my favorite solution so far, it works great and is incredible at searching for partial words and wrong spellings. I have not had an opportunity to use it in production yet, but I'm hoping to install it on one of my forums later this year.

CyberGene

I've been using Meilisearch through this extension for some time now and I'm very happy with it. The only thing is I have to pay €7 a month for the DigitalOcean droplet and while that's not much, I'm still not sure if it's worth it over the Flarum native search. I'll have to check with my users about how often they use the search functionality but it seems they don't care a lot and don't use it too much, so I may switch to Flarum native again unless I find cheaper Meilisearch solution since my current needs are just too tiny to justify even €7 a month.

clarkwinkelmann

CyberGene where are you hosting your forum? You might be able to place Flarum and Meilisearch on the same server to save on cost.

Meilisearch seem to be working on a cloud hosting, but there's just a waitlist and no mention of the pricing. I doubt it would be affordable for this kind of use case but we never know.

It's true that if few people use the search bar, switching driver might not make a lot of sense. One place where people might not immediately realize is using the search are mentions. The auto-complete relies on the same user search endpoint. But user search is also not dramatically improved over the native search since it's just usernames that are indexed anyway.

CyberGene

clarkwinkelmann I’m hosting my Flarum on Siteground. I could use another hosting where I have more choice about the cloud services but I will have to research it first. Siteground has been very reliable and fast, especially since I was on A2 before which was very unreliable.

zjmw00

So I installed everything and did a php flarum scout:import-all and it ran through all the ID's, but now search returns nothing.

I'm not seeing any output from meilisearch in the console it's running in when importing or performing a search either, although I'm not sure I'm supposed to. What am I doing wrong?

I think it may not be populating the indexes with any data, but I'm not entirely sure how to verify that. I've tried sending some GET requests with curl but I can't seem to get anything back.

clarkwinkelmann

zjmw00 you can access meilisearch web dashboard (open meilisearch host:post in your browser) and check that the index contains entries and use the dashboard search to see if you get any result.

If everything works as expected in meilisearch web dashboard, I'm not sure what it could be... Can you share your php flarum info output and some more details about your hosting situation and the way you installed meilisearch?

zjmw00

clarkwinkelmann
In the web UI it shows no indexes found. I assume the import isn't working. I have around 5 million posts, and it took quite a while for that import to run. Visually, it looks like it's doing something, but I guess it wasn't. The import showed it importing users, discussions, and posts, if I remember correctly.

Here's php flarum info

Flarum core 1.5.0
PHP version: 7.4.32
MySQL version: 5.5.5-10.6.7-MariaDB-2ubuntu1.1
Loaded extensions: Core, date, libxml, openssl, pcre, zlib, filter, hash, pcntl, Reflection, SPL, session, standard, sodium, mysqlnd, PDO, xml, apcu, bcmath, calendar, ctype, curl, dom, mbstring, FFI, fileinfo, ftp, gd, gettext, iconv, igbinary, imagick, imap, intl, json, exif, memcache, msgpack, mysqli, pdo_mysql, Phar, posix, readline, redis, shmop, SimpleXML, sockets, sysvmsg, sysvsem, sysvshm, tokenizer, xmlreader, xmlrpc, xmlwriter, xsl, zip, memcached, Zend OPcache
+-----------------------+---------+--------+
| Flarum Extensions     |         |        |
+-----------------------+---------+--------+
| ID                    | Version | Commit |
+-----------------------+---------+--------+
| flarum-flags          | v1.5.0  |        |
| flarum-tags           | v1.5.0  |        |
| flarum-suspend        | v1.5.0  |        |
| flarum-subscriptions  | v1.5.0  |        |
| flarum-sticky         | v1.5.0  |        |
| flarum-statistics     | v1.5.0  |        |
| flarum-mentions       | v1.5.0  |        |
| flarum-markdown       | v1.5.0  |        |
| flarum-lock           | v1.5.0  |        |
| flarum-likes          | v1.5.0  |        |
| flarum-lang-english   | v1.5.0  |        |
| flarum-emoji          | v1.5.0  |        |
| flarum-bbcode         | v1.5.0  |        |
| flarum-approval       | v1.5.0  |        |
| clarkwinkelmann-scout | 0.2.0   |        |
+-----------------------+---------+--------+
Base URL: https://______________.com
Installation path: /var/www/archive
Queue driver: sync
Mail driver: mail
Debug mode: off

I installed Meilisearch following this:
https://docs.meilisearch.com/learn/cookbooks/running_production.html#step-1-install-meilisearch

I stopped when I got to creating a service for it, I'll do that eventually but for now I'm just running the binary in a screen session.

One thing I've noticed is, when I use the scout commands such as php flarum scout:index testindex, there's no web request being made to Meilisearch, same with when I import indexes with php flarum scout:import-all.

Meilisearch is installed locally to Flarum.

Does Scout expect Meilisearch to be configured with https? Maybe that's why it doesn't hit it. I didn't go that far.

clarkwinkelmann

zjmw00 I only tested with mailisearch http, so that should be fine.

In the extension settings, is the driver choice correctly on meilisearch? It's strange if zero requests are made to meilisearch at all...