Scout Search for Flarum

MIT license Latest Stable Version Total Downloads Donate

Integrates Laravel Scout with Flarum discussion and user search.

Just like with Laravel, the data is automatically synced with the search index every time a model is updated in Flarum.
You only need to manually import data when you enable the extension (see commands below).

The external search driver is used server-side to filter down the MySQL results, so it should still be compatible with every other extension and search gambits.

Algolia and Meilisearch drivers are included in the extension.
TNTSearch is supported but requires the manual installation of an additional package.
The Scout database and collection drivers cannot be used (they would be worst than Flarum's built-in database search).

See below for the specific requirements and configuration of each driver.

While only discussions and users are searchable in Flarum, this implementation also uses a posts search index which is merged with discussion search results in a similar way to the Flarum native search.
The discussion result sort currently prioritize best post matching because I have not found a way to merge the match score of discussions and posts indices.

All CLI commands from Scout are available, with an additional special "import all" command:

php flarum scout:import-all           Import all Flarum models into the search index
                                      (a shortcut to scout:import with every searchable class known to Flarum)
php flarum scout:flush {model}        Flush all of the model's records from the index
php flarum scout:import {model}       Import the given model into the search index
php flarum scout:index {name}         Create an index (generally not needed)
php flarum scout:delete-index {name}  Delete an index (generally not needed)

Algolia

The Algolia driver requires an account on the eponymous cloud service.

Meilisearch

The Meilisearch driver requires a running Meilisearch server instance.
The server can be hosted anywhere as long as it can be reached over the network.
By default, the extension attempts to connect to a server at 127.0.0.1:7700.

If you are not running the latest Meilisearch version you might need to explicitly install an older version of the SDK.
Likewise, if you are regularly running composer update on all your dependencies, you should also add an explicit requirement for the Meilisearch SDK in your composer.json because the extension requires * which might jump to a newer Meilisearch SDK version as soon as it comes out.

To install and lock the current latest version:

composer require meilisearch/meilisearch-php

Unfortunately Meilisearch doesn't seem to advertise which specific version of the Composer package is compatible with each server version.
You can find the list of releases at https://packagist.org/packages/meilisearch/meilisearch-php

Once you know which version you need, you can lock it, for example to install the older 0.23:

composer require meilisearch/meilisearch-php:"0.23.*"

The only settings for Meilisearch are Host and Key.
Everything else is configured in the Meilisearch server itself.

Even if you don't configure the Default Results Limit value and use Meilisearch, the extension will automatically set it to 200 for you because the default for Meilisearch (20) is extremely low and result in only 2 pages of results at best.

TNTSearch

The TNTSearch library requires the sqlite PHP extension, therefore it's not included by default with Scout.

To install it, make sure you have the sqlite PHP extension enabled for both command line and webserver and run:

composer require teamtnt/laravel-scout-tntsearch-driver

TNTSearch uses local sqlite databases for each index.
The databases are stored in <flarum>/storage/tntsearch which must be writable.

The following settings are exposed.
What each setting does isn't entirely clear, TNTSearch own documentation doesn't offer much guidance.

  • Max Docs: this likely impacts how many results Flarum will be able to show for a query
  • Fuziness (on/off): seems to be the typos/variation matching
  • Fuzziness Levenshtein Distance
  • Fuzziness Prefix Length: no idea what it does
  • Fuzziness Max Expansions no idea what it does

As You Type and Search Boolean are hard-coded to enabled, though they don't seem to work as described in TNTSearch documentation.

Installation

This extension is still experimental. Please test on a staging server first.
I have not tested Algolia personally yet, but from feedback it seems to be working.

composer require clarkwinkelmann/flarum-ext-scout

Supported extensions and fields

This list is not exhaustive.
If you added support for Scout in your extension, let me know so I can update this list.

Discussions

When searching for discussions via Flarum's search feature, the fields for Posts are also queried.

  • Title: support built into Scout.
  • Formulaire Discussion Fields: supported since Formulaire 1.8 (guest-accessible forms only).

Posts

  • Content: support built into Scout, the value used is a plain text version of the output HTML without any tag. Some information like link URLs, image URLs and image alts are therefore not indexed. This might be changed in a future version.

Users

  • Display Name: support built into Scout.
  • Username: support built into Scout.
  • FoF Bio: support built into Scout (not in FoF Bio itself).
  • Formulaire Profile Fields: supported since Formulaire 1.8 (guest-accessible forms only).

Email is intentionally not searchable because there's currently no mechanism that would prevent regular users from using that feature to leak email.

Formulaire

Forms and Submissions are optionally indexed via Scout. See Formulaire documentation for details.

Developers

See README on GitHub for up to date information https://github.com/clarkwinkelmann/flarum-ext-scout#developers

Support

This extension is under minimal maintenance.

It was developed for a client and released as open-source for the benefit of the community.
I might publish simple bugfixes or compatibility updates for free.

You can contact me to sponsor additional features or updates.

Support is offered on a "best effort" basis through the Flarum community thread.

Links

    mistermochi thanks for sharing the installation steps

    clarkwinkelmann We want algolia search feature because we have multiple subdomains and subdirectories with different technologies (flarum, wordpress). We thought of building one single centralized search page like CF search I think if we use algolia it would be an easy approach if you have any other suggestions do suggest.

      Hari I'm not super familiar with Algolia so not really sure about the best approach here.

      I don't think you can use a single unified index for Flarum stuff and another app, as scout requires its own dedicated index for each resource. But maybe Algolia has ways to merge data from multiple indexes when performing a search via the API, not sure.

        a month later

        clarkwinkelmann tried on flarumde for fun with Algolia. Importing the data to Algolia worked like a charm But I am receiving 500 errors when trying to search.

        request.js:170 GET https://www.flarumde.com/api/discussions?filter%5Bq%5D=ank&page%5Blimit%5D=3&include=mostRelevantPost 500 (Internal Server Error)

        Next Illuminate\Database\QueryException: SQLSTATE[42S22]: Column not found: 1054 Unknown column 'matching_posts.priority' in 'field list' (SQL: select `flarumde_discussions`.*, COALESCE(posts_ft.most_relevant_post_id, discussions.first_post_id) as most_relevant_post_id from `flarumde_discussions` left join (select `flarumde_posts`.`discussion_id`, id as most_relevant_post_id from `flarumde_posts` inner join (select `flarumde_posts`.`discussion_id`, min(matching_posts.priority) as min_priority from `flarumde_posts` inner join (select `flarumde_posts`.`discussion_id`, FIELD(id, 67, 179, 209, 161, 7) as priority from `flarumde_posts` where exists (select 1 from `flarumde_discussions` where `flarumde_discussions`.`id` = `flarumde_posts`.`discussion_id` and (`flarumde_discussions`.`id` not in (select `discussion_id` from `flarumde_discussion_tag` where `tag_id` not in (select `flarumde_tags`.`id` from `flarumde_tags` where (`flarumde_tags`.`id` in (select `flarumde_perm_tags`.`id` from `flarumde_tags` as `flarumde_perm_tags`) and (`flarumde_tags`.`parent_id` in (select `flarumde_perm_tags`.`id` from `flarumde_tags` as `flarumde_perm_tags`) or `flarumde_tags`.`parent_id` is null))))) and (`flarumde_discussions`.`is_private` = 0 or (((`flarumde_discussions`.`is_approved` = 0) or (`flarumde_discussions`.`id` in (select distinct `flarumde_recipients`.`discussion_id` from `flarumde_recipients` where `flarumde_recipients`.`removed_at` is null and (`flarumde_recipients`.`user_id` = 1 or `flarumde_recipients`.`group_id` in (1, 5))) or (`flarumde_discussions`.`id` in (select `flarumde_recipients`.`discussion_id` from `flarumde_recipients` where `flarumde_recipients`.`removed_at` is null) and `flarumde_discussions`.`id` in (select distinct `flarumde_posts`.`discussion_id` from `flarumde_posts` where `flarumde_posts`.`id` in (select distinct `flarumde_flags`.`post_id` from `flarumde_flags`)))))))) and (`flarumde_posts`.`is_private` = 0 or (((`flarumde_posts`.`is_approved` = 0)))) and `flarumde_posts`.`type` = comment and `id` in (67, 179, 209, 161, 7) and `type` in (discussionTagged, discussionSplit, discussionStickied, discussionLocked, recipientLeft, recipientsModified, madePublic, discussionMerged, comment, discussionRenamed)) `flarumde_matching_posts` on `flarumde_matching_posts`.`discussion_id` = `flarumde_posts`.`discussion_id` where `type` in (discussionTagged, discussionSplit, discussionStickied, discussionLocked, recipientLeft, recipientsModified, madePublic, discussionMerged, comment, discussionRenamed) group by `flarumde_posts`.`discussion_id`) `flarumde_best_matching_posts` on `flarumde_best_matching_posts`.`discussion_id` = `flarumde_posts`.`discussion_id` where exists (select 1 from `flarumde_discussions` where `flarumde_discussions`.`id` = `flarumde_posts`.`discussion_id` and (`flarumde_discussions`.`id` not in (select `discussion_id` from `flarumde_discussion_tag` where `tag_id` not in (select `flarumde_tags`.`id` from `flarumde_tags` where (`flarumde_tags`.`id` in (select `flarumde_perm_tags`.`id` from `flarumde_tags` as `flarumde_perm_tags`) and (`flarumde_tags`.`parent_id` in (select `flarumde_perm_tags`.`id` from `flarumde_tags` as `flarumde_perm_tags`) or `flarumde_tags`.`parent_id` is null))))) and (`flarumde_discussions`.`is_private` = 0 or (((`flarumde_discussions`.`is_approved` = 0) or (`flarumde_discussions`.`id` in (select distinct `flarumde_recipients`.`discussion_id` from `flarumde_recipients` where `flarumde_recipients`.`removed_at` is null and (`flarumde_recipients`.`user_id` = 1 or `flarumde_recipients`.`group_id` in (1, 5))) or (`flarumde_discussions`.`id` in (select `flarumde_recipients`.`discussion_id` from `flarumde_recipients` where `flarumde_recipients`.`removed_at` is null) and `flarumde_discussions`.`id` in (select distinct `flarumde_posts`.`discussion_id` from `flarumde_posts` where `flarumde_posts`.`id` in (select distinct `flarumde_flags`.`post_id` from `flarumde_flags`)))))))) and (`flarumde_posts`.`is_private` = 0 or (((`flarumde_posts`.`is_approved` = 0)))) and `id` in (67, 179, 209, 161, 7) and FIELD(id, 67, 179, 209, 161, 7) = best_matching_posts.min_priority and `type` in (discussionTagged, discussionSplit, discussionStickied, discussionLocked, recipientLeft, recipientsModified, madePublic, discussionMerged, comment, discussionRenamed)) `flarumde_posts_ft` on `flarumde_posts_ft`.`discussion_id` = `flarumde_discussions`.`id` where (`flarumde_discussions`.`id` not in (select `discussion_id` from `flarumde_discussion_tag` where `tag_id` not in (select `flarumde_tags`.`id` from `flarumde_tags` where (`flarumde_tags`.`id` in (select `flarumde_perm_tags`.`id` from `flarumde_tags` as `flarumde_perm_tags`) and (`flarumde_tags`.`parent_id` in (select `flarumde_perm_tags`.`id` from `flarumde_tags` as `flarumde_perm_tags`) or `flarumde_tags`.`parent_id` is null))))) and (`flarumde_discussions`.`is_private` = 0 or (((`flarumde_discussions`.`is_approved` = 0) or (`flarumde_discussions`.`id` in (select distinct `flarumde_recipients`.`discussion_id` from `flarumde_recipients` where `flarumde_recipients`.`removed_at` is null and (`flarumde_recipients`.`user_id` = 1 or `flarumde_recipients`.`group_id` in (1, 5))) or (`flarumde_discussions`.`id` in (select `flarumde_recipients`.`discussion_id` from `flarumde_recipients` where `flarumde_recipients`.`removed_at` is null) and `flarumde_discussions`.`id` in (select distinct `flarumde_posts`.`discussion_id` from `flarumde_posts` where `flarumde_posts`.`id` in (select distinct `flarumde_flags`.`post_id` from `flarumde_flags`))))))) and (`most_relevant_post_id` is not null or 0 = 1) group by `flarumde_discussions`.`id` order by FIELD(id, 67, 179, 209, 161, 7) limit 4) in /path/httpdocs/vendor/illuminate/database/Connection.php:712
        Stack trace:
        #0 /path/httpdocs/vendor/illuminate/database/Connection.php(672): Illuminate\Database\Connection->runQueryCallback()
        #1 /path/httpdocs/vendor/illuminate/database/Connection.php(376): Illuminate\Database\Connection->run()
        #2 /path/httpdocs/vendor/illuminate/database/Query/Builder.php(2414): Illuminate\Database\Connection->select()
        #3 /path/httpdocs/vendor/illuminate/database/Query/Builder.php(2402): Illuminate\Database\Query\Builder->runSelect()
        #4 /path/httpdocs/vendor/illuminate/database/Query/Builder.php(2936): Illuminate\Database\Query\Builder->Illuminate\Database\Query\{closure}()
        #5 /path/httpdocs/vendor/illuminate/database/Query/Builder.php(2403): Illuminate\Database\Query\Builder->onceWithColumns()
        #6 /path/httpdocs/vendor/illuminate/database/Eloquent/Builder.php(625): Illuminate\Database\Query\Builder->get()
        #7 /path/httpdocs/vendor/illuminate/database/Eloquent/Builder.php(609): Illuminate\Database\Eloquent\Builder->getModels()
        #8 /path/httpdocs/vendor/flarum/core/src/Search/AbstractSearcher.php(68): Illuminate\Database\Eloquent\Builder->get()
        #9 /path/httpdocs/vendor/flarum/core/src/Api/Controller/ListDiscussionsController.php(100): Flarum\Search\AbstractSearcher->search()
        #10 /path/httpdocs/vendor/flarum/core/src/Api/Controller/AbstractSerializeController.php(115): Flarum\Api\Controller\ListDiscussionsController->data()
        #11 /path/httpdocs/vendor/flarum/core/src/Http/RouteHandlerFactory.php(41): Flarum\Api\Controller\AbstractSerializeController->handle()
        #12 /path/httpdocs/vendor/flarum/core/src/Http/Middleware/ExecuteRoute.php(27): Flarum\Http\RouteHandlerFactory->Flarum\Http\{closure}()
        #13 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Flarum\Http\Middleware\ExecuteRoute->process()
        #14 /path/httpdocs/vendor/kilowhat/flarum-ext-audit-pro/src/Middlewares/ExtendSetPermissionController.php(19): Laminas\Stratigility\Next->handle()
        #15 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Kilowhat\Audit\Middlewares\ExtendSetPermissionController->process()
        #16 /path/httpdocs/vendor/kilowhat/flarum-ext-audit-pro/src/Middlewares/SetLoggerActor.php(28): Laminas\Stratigility\Next->handle()
        #17 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Kilowhat\Audit\Middlewares\SetLoggerActor->process()
        #18 /path/httpdocs/vendor/flarum/core/src/Api/Middleware/ThrottleApi.php(33): Laminas\Stratigility\Next->handle()
        #19 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Flarum\Api\Middleware\ThrottleApi->process()
        #20 /path/httpdocs/vendor/flarum/core/src/Http/Middleware/CheckCsrfToken.php(36): Laminas\Stratigility\Next->handle()
        #21 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Flarum\Http\Middleware\CheckCsrfToken->process()
        #22 /path/httpdocs/vendor/flarum/core/src/Http/Middleware/ResolveRoute.php(67): Laminas\Stratigility\Next->handle()
        #23 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Flarum\Http\Middleware\ResolveRoute->process()
        #24 /path/httpdocs/vendor/flarum/core/src/Http/Middleware/SetLocale.php(51): Laminas\Stratigility\Next->handle()
        #25 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Flarum\Http\Middleware\SetLocale->process()
        #26 /path/httpdocs/vendor/flarum/core/src/Http/Middleware/AuthenticateWithHeader.php(58): Laminas\Stratigility\Next->handle()
        #27 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Flarum\Http\Middleware\AuthenticateWithHeader->process()
        #28 /path/httpdocs/vendor/flarum/core/src/Http/Middleware/AuthenticateWithSession.php(31): Laminas\Stratigility\Next->handle()
        #29 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Flarum\Http\Middleware\AuthenticateWithSession->process()
        #30 /path/httpdocs/vendor/flarum/core/src/Http/Middleware/RememberFromCookie.php(52): Laminas\Stratigility\Next->handle()
        #31 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Flarum\Http\Middleware\RememberFromCookie->process()
        #32 /path/httpdocs/vendor/flarum/core/src/Http/Middleware/StartSession.php(61): Laminas\Stratigility\Next->handle()
        #33 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Flarum\Http\Middleware\StartSession->process()
        #34 /path/httpdocs/vendor/flarum/core/src/Api/Middleware/FakeHttpMethods.php(29): Laminas\Stratigility\Next->handle()
        #35 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Flarum\Api\Middleware\FakeHttpMethods->process()
        #36 /path/httpdocs/vendor/flarum/core/src/Http/Middleware/ParseJsonBody.php(28): Laminas\Stratigility\Next->handle()
        #37 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Flarum\Http\Middleware\ParseJsonBody->process()
        #38 /path/httpdocs/vendor/flarum/core/src/Http/Middleware/HandleErrors.php(57): Laminas\Stratigility\Next->handle()
        #39 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Flarum\Http\Middleware\HandleErrors->process()
        #40 /path/httpdocs/vendor/flarum/core/src/Http/Middleware/InjectActorReference.php(25): Laminas\Stratigility\Next->handle()
        #41 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Flarum\Http\Middleware\InjectActorReference->process()
        #42 /path/httpdocs/vendor/laminas/laminas-stratigility/src/MiddlewarePipe.php(76): Laminas\Stratigility\Next->handle()
        #43 /path/httpdocs/vendor/middlewares/request-handler/src/RequestHandler.php(84): Laminas\Stratigility\MiddlewarePipe->process()
        #44 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Middlewares\RequestHandler->process()
        #45 /path/httpdocs/vendor/middlewares/base-path-router/src/BasePathRouter.php(101): Laminas\Stratigility\Next->handle()
        #46 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Middlewares\BasePathRouter->process()
        #47 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Middleware/OriginalMessages.php(36): Laminas\Stratigility\Next->handle()
        #48 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Laminas\Stratigility\Middleware\OriginalMessages->process()
        #49 /path/httpdocs/vendor/middlewares/base-path/src/BasePath.php(73): Laminas\Stratigility\Next->handle()
        #50 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Middlewares\BasePath->process()
        #51 /path/httpdocs/vendor/flarum/core/src/Http/Middleware/ProcessIp.php(24): Laminas\Stratigility\Next->handle()
        #52 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Flarum\Http\Middleware\ProcessIp->process()
        #53 /path/httpdocs/vendor/laminas/laminas-stratigility/src/MiddlewarePipe.php(76): Laminas\Stratigility\Next->handle()
        #54 /path/httpdocs/vendor/laminas/laminas-stratigility/src/MiddlewarePipe.php(65): Laminas\Stratigility\MiddlewarePipe->process()
        #55 /path/httpdocs/vendor/laminas/laminas-httphandlerrunner/src/RequestHandlerRunner.php(96): Laminas\Stratigility\MiddlewarePipe->handle()
        #56 /path/httpdocs/vendor/flarum/core/src/Http/Server.php(44): Laminas\HttpHandlerRunner\RequestHandlerRunner->run()
        #57 /path/httpdocs/public/index.php(26): Flarum\Http\Server->listen()
        #58 {main}  
        [2022-06-27 11:57:31] flarum.ERROR: PDOException: SQLSTATE[42S22]: Column not found: 1054 Unknown column 'matching_posts.priority' in 'field list' in /path/httpdocs/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOConnection.php:82
        Stack trace:
        #0 /path/httpdocs/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOConnection.php(82): PDO->prepare()
        #1 /path/httpdocs/vendor/illuminate/database/Connection.php(368): Doctrine\DBAL\Driver\PDOConnection->prepare()
        #2 /path/httpdocs/vendor/illuminate/database/Connection.php(705): Illuminate\Database\Connection->Illuminate\Database\{closure}()
        #3 /path/httpdocs/vendor/illuminate/database/Connection.php(672): Illuminate\Database\Connection->runQueryCallback()
        #4 /path/httpdocs/vendor/illuminate/database/Connection.php(376): Illuminate\Database\Connection->run()
        #5 /path/httpdocs/vendor/illuminate/database/Query/Builder.php(2414): Illuminate\Database\Connection->select()
        #6 /path/httpdocs/vendor/illuminate/database/Query/Builder.php(2402): Illuminate\Database\Query\Builder->runSelect()
        #7 /path/httpdocs/vendor/illuminate/database/Query/Builder.php(2936): Illuminate\Database\Query\Builder->Illuminate\Database\Query\{closure}()
        #8 /path/httpdocs/vendor/illuminate/database/Query/Builder.php(2403): Illuminate\Database\Query\Builder->onceWithColumns()
        #9 /path/httpdocs/vendor/illuminate/database/Eloquent/Builder.php(625): Illuminate\Database\Query\Builder->get()
        #10 /path/httpdocs/vendor/illuminate/database/Eloquent/Builder.php(609): Illuminate\Database\Eloquent\Builder->getModels()
        #11 /path/httpdocs/vendor/flarum/core/src/Search/AbstractSearcher.php(68): Illuminate\Database\Eloquent\Builder->get()
        #12 /path/httpdocs/vendor/flarum/core/src/Api/Controller/ListDiscussionsController.php(100): Flarum\Search\AbstractSearcher->search()
        #13 /path/httpdocs/vendor/flarum/core/src/Api/Controller/AbstractSerializeController.php(115): Flarum\Api\Controller\ListDiscussionsController->data()
        #14 /path/httpdocs/vendor/flarum/core/src/Http/RouteHandlerFactory.php(41): Flarum\Api\Controller\AbstractSerializeController->handle()
        #15 /path/httpdocs/vendor/flarum/core/src/Http/Middleware/ExecuteRoute.php(27): Flarum\Http\RouteHandlerFactory->Flarum\Http\{closure}()
        #16 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Flarum\Http\Middleware\ExecuteRoute->process()
        #17 /path/httpdocs/vendor/kilowhat/flarum-ext-audit-pro/src/Middlewares/ExtendSetPermissionController.php(19): Laminas\Stratigility\Next->handle()
        #18 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Kilowhat\Audit\Middlewares\ExtendSetPermissionController->process()
        #19 /path/httpdocs/vendor/kilowhat/flarum-ext-audit-pro/src/Middlewares/SetLoggerActor.php(28): Laminas\Stratigility\Next->handle()
        #20 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Kilowhat\Audit\Middlewares\SetLoggerActor->process()
        #21 /path/httpdocs/vendor/flarum/core/src/Api/Middleware/ThrottleApi.php(33): Laminas\Stratigility\Next->handle()
        #22 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Flarum\Api\Middleware\ThrottleApi->process()
        #23 /path/httpdocs/vendor/flarum/core/src/Http/Middleware/CheckCsrfToken.php(36): Laminas\Stratigility\Next->handle()
        #24 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Flarum\Http\Middleware\CheckCsrfToken->process()
        #25 /path/httpdocs/vendor/flarum/core/src/Http/Middleware/ResolveRoute.php(67): Laminas\Stratigility\Next->handle()
        #26 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Flarum\Http\Middleware\ResolveRoute->process()
        #27 /path/httpdocs/vendor/flarum/core/src/Http/Middleware/SetLocale.php(51): Laminas\Stratigility\Next->handle()
        #28 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Flarum\Http\Middleware\SetLocale->process()
        #29 /path/httpdocs/vendor/flarum/core/src/Http/Middleware/AuthenticateWithHeader.php(58): Laminas\Stratigility\Next->handle()
        #30 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Flarum\Http\Middleware\AuthenticateWithHeader->process()
        #31 /path/httpdocs/vendor/flarum/core/src/Http/Middleware/AuthenticateWithSession.php(31): Laminas\Stratigility\Next->handle()
        #32 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Flarum\Http\Middleware\AuthenticateWithSession->process()
        #33 /path/httpdocs/vendor/flarum/core/src/Http/Middleware/RememberFromCookie.php(52): Laminas\Stratigility\Next->handle()
        #34 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Flarum\Http\Middleware\RememberFromCookie->process()
        #35 /path/httpdocs/vendor/flarum/core/src/Http/Middleware/StartSession.php(61): Laminas\Stratigility\Next->handle()
        #36 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Flarum\Http\Middleware\StartSession->process()
        #37 /path/httpdocs/vendor/flarum/core/src/Api/Middleware/FakeHttpMethods.php(29): Laminas\Stratigility\Next->handle()
        #38 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Flarum\Api\Middleware\FakeHttpMethods->process()
        #39 /path/httpdocs/vendor/flarum/core/src/Http/Middleware/ParseJsonBody.php(28): Laminas\Stratigility\Next->handle()
        #40 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Flarum\Http\Middleware\ParseJsonBody->process()
        #41 /path/httpdocs/vendor/flarum/core/src/Http/Middleware/HandleErrors.php(57): Laminas\Stratigility\Next->handle()
        #42 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Flarum\Http\Middleware\HandleErrors->process()
        #43 /path/httpdocs/vendor/flarum/core/src/Http/Middleware/InjectActorReference.php(25): Laminas\Stratigility\Next->handle()
        #44 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Flarum\Http\Middleware\InjectActorReference->process()
        #45 /path/httpdocs/vendor/laminas/laminas-stratigility/src/MiddlewarePipe.php(76): Laminas\Stratigility\Next->handle()
        #46 /path/httpdocs/vendor/middlewares/request-handler/src/RequestHandler.php(84): Laminas\Stratigility\MiddlewarePipe->process()
        #47 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Middlewares\RequestHandler->process()
        #48 /path/httpdocs/vendor/middlewares/base-path-router/src/BasePathRouter.php(101): Laminas\Stratigility\Next->handle()
        #49 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Middlewares\BasePathRouter->process()
        #50 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Middleware/OriginalMessages.php(36): Laminas\Stratigility\Next->handle()
        #51 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Laminas\Stratigility\Middleware\OriginalMessages->process()
        #52 /path/httpdocs/vendor/middlewares/base-path/src/BasePath.php(73): Laminas\Stratigility\Next->handle()
        #53 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Middlewares\BasePath->process()
        #54 /path/httpdocs/vendor/flarum/core/src/Http/Middleware/ProcessIp.php(24): Laminas\Stratigility\Next->handle()
        #55 /path/httpdocs/vendor/laminas/laminas-stratigility/src/Next.php(51): Flarum\Http\Middleware\ProcessIp->process()
        #56 /path/httpdocs/vendor/laminas/laminas-stratigility/src/MiddlewarePipe.php(76): Laminas\Stratigility\Next->handle()
        #57 /path/httpdocs/vendor/laminas/laminas-stratigility/src/MiddlewarePipe.php(65): Laminas\Stratigility\MiddlewarePipe->process()
        #58 /path/httpdocs/vendor/laminas/laminas-httphandlerrunner/src/RequestHandlerRunner.php(96): Laminas\Stratigility\MiddlewarePipe->handle()
        #59 /path/httpdocs/vendor/flarum/core/src/Http/Server.php(44): Laminas\HttpHandlerRunner\RequestHandlerRunner->run()
        #60 /path/httpdocs/public/index.php(26): Flarum\Http\Server->listen()
        #61 {main}

          I installed it, imported the posts into the index and the search even works great, matching partial search terms...

          However I am getting 500 while browsing the forum 😕

          OK, I got it, in the Flarum logs there's Algolia message saying I exceeded the quota (10k objects and we have 11k posts 😀)... So, I upgraded to the $1 a unit tier, let's see how much it will cost, my understanding is $1 for each additional 1000 objects.

            Seems to be working well so far, thanks a lot! Much better than the embedded SQL search.

            I see some odd situations though. For instance I created a post where I have a sentence containing "Chinese manufactured". If I search for "Chinese", I get results from other discussions and posts but not this one. If I search for "Chinese manufactured", then it's found. A bit odd, why would that happen?

              GreXXL it seems to be an issue caused by the use of a database table prefix. I'll look into this right away.

              CyberGene If I search for "Chinese", I get results from other discussions and posts but not this one

              Yep, that's odd. I haven't tested Algolia yet so not sure what's happening. I'll try replicating with Meilisearch this afternoon and see if it gives me any clue.

              I'll need to do a proper and complete Algolia test at some point but haven't had the time yet.

              CyberGene I exceeded the quota

              Yep, every post (including first post of discussions), every discussion and every user will most likely count as one object. There's currently no way to exclude a specific resource but I could imagine adding an option to exclude users from Scout if this helps reduce service cost and isn't absolutely needed.

              Version 0.1.2

              • Fixed support for using Flarum with a database prefix
              • Fixed posts not syncing to index when created through a discussion or a reply
              • Fixed old content being synced when a post is edited

              I recommend running the scout:import-all command to fix all existing broken records.

              This update changes the way post content is indexed. Previously the markdown/bbcode version of the post from edit mode would be sent to the index. Now it's the rendered HTML version except it's stripped of any HTML. This should prevent hidden information from being leaked via search (example: old username if a mentioned user changed their username, content hidden with the hide bbcode/reply/likes extensions). But it also means image and link URLs are now no longer indexed. I will try to add an option to get those URLs indexed again in a next version.


              CyberGene I was not able to replicate your issue on Meilisearch, however I found out the synchronization wasn't working correctly. So maybe your new post wasn't indexed yet.

              If the problem happens again, can you try performing the search directly in Algolia's dashboard? To see if any result exists in the search index? Meilisearch has a nice built-in web interface that allows inspecting the current state of the index and I believe Algolia has something similar.

                clarkwinkelmann thanks, I installed the update and reimported the index. Seems like the search issue I had is fixed now, so maybe the problem was with what you described as possible reason. I will let you know if there are some other issues. So far all looks pretty good, thanks a lot! 🤝

                Hari Thanks, I may check that later after I see how much Algolia will cost me this month.

                Hari what's this digital ocean service, is it verified? I'm trying to register myself and can't go through the initial payment validation. I tried every possible payment method: Apple Pay, PayPal, direct credit card with all my cards and nothing goes through, they are all rejected for some reason. I'm giving up...

                • Hari replied to this.

                  Never mind, registered with Revolut, maybe my bank is having some problems currently.

                  CyberGene as far i know you need not to pay any money to set-up a $5 droplet 🤔 they do accept debit cards too. And you get $100 free credit as a sign-up bonus (not sure about this as they advertise this)

                    clarkwinkelmann If you are regularly running composer update on all your dependencies, you should add an explicit requirement for the Meilisearch SDK in your composer.json because the extension requires * which might jump to a newer Meilisearch SDK version as soon as it comes out.

                    Can you tell me what exact stuff should I put in the composer.json?

                    I switched to Meilisearch in DigitalOcean, as explained by the link Hari posted. I imported the indexes, all works. I guess I will use it instead of Algolia because it's cheaper. Thanks Hari

                      a month later

                      Hello, I use the extension with meili search and it works well, thank you!
                      However I find that the results that match content is prior to those match title. How shall I change the priority?
                      Thank you!

                        tohsakarat unfortunately it's a consequence of the current implementation, so can't be changed.

                        I'm not sure what the best way to calculate the sort order is. The problem is there are 2 indexes, one for discussion data (title) and one for post data (content). And I can't get the Meilisearch matching score out of Scout, only the order of the results. So merging the 2 in a way that makes sense is quite challenging. I guess it will require some specific code to go deeper level into Meilisearch API or modify the logic to use a single index.

                        24 days later

                        Version 0.2.0 - TNTSearch

                        Following a suggestion by @010101 I have added TNTSearch support to the extension.

                        Existing users don't need to install this update if they are not planning to try TNTSearch.

                        To update:

                        composer require clarkwinkelmann/flarum-ext-scout

                        Additionally, to use TNTSearch you need to install an additional package. This package requires the sqlite PHP extension on the server!

                        composer require teamtnt/laravel-scout-tntsearch-driver

                        See the README on GitHub for more information https://github.com/clarkwinkelmann/flarum-ext-scout#tntsearch

                        If you switch driver, you need to rebuild the indexes with the command!

                        From my tests, TNTSearch seems an incredibly poor search engine. Their documentation is very lacking and the few options/syntaxes that are described don't seem to work. Maybe it can be improved slightly by playing with the settings, but I'm not sure it brings much improvement over Flarum native search.

                        This extension is still very experimental and designed to test alternate search engines with the goal of maybe integrating some of them natively in Flarum in the future.