Blomstra Search, an elasticsearch fulltext search extension

maisen20

Do I understand correctly that this search function does not search in discussion titles?

luceos

maisen20 it can. It's an option in the extension settings.

maisen20

luceos okay but i have some issues. for example i have a discussion with title: "Arbeitsstunden, Ferientage, Feiertage und Ferienbonus definieren".

Now when I search for "Arbeitsstu", the discussion is suggested as a result. But as soon as I search for "Arbeitsstun" (one letter more at the end), no result is displayed.

What could be the reason for this?

luceos

maisen20 did you select the right language? And you might need to rerun seeding after that.

maisen20

The language is set correctly (German) and I have already deleted the index "flarum_forum" completely and created it again.

It seems that it does not find results when my search string is longer than 10 characters.

luceos

maisen20 I don't see this issue in production: https://forum.mutluanneleriz.com/?q=hastane%20%C3%A7antas%C4%B1n%C4%B1%20haz%C4%B1rlamaya%20eksik%20hep%20%C3%A7%C4%B1k%C4%B1yor%20%C3%A7%C3%BCnk%C3%BC%20birde

Can you try identifying what is sent through the Flarum api?

maisen20

When i check the console i can see the request url is:

/api/blomstra/search/discussions?filter[q]=filtervorlagen&page[limit]=3&include=mostRelevantPost

I can't see errors in the logs and the result is:

{"links":{"first":"https:\/\/community.domain.com\/api\/blomstra\/search\/discussions?filter%5Bq%5D=filtervorlagen\u0026page%5Blimit%5D=20\u0026include=mostRelevantPost\u0026type=discussions"},"data":[]}

luceos

maisen20 please reach out to me on discord (my dm's are open if you joined Flarum discord) so you can share your url privately and I can take a look. That's much faster than this back and forth 😉

maisen20

To inform everyone:

I figured out what the reason for the limitation to 10 characters is.
In the file "/vendor/blomstra/search/src/Commands/BuildCommand.php" there are the values for 'index.max_ngram_diff' and 'max_gram' which tell ES's tokenizer how long the token can be. These values are set to 10. I assume that this is the case for all of you.

This means words containing more than 10 characters will not be tokenized and therefore these words/posts will not be found.

Mirzaagha

composer require blomstra/search:"*"

l want install this extension but write a error words

luceos

Mirzaagha try running the command with -W

Mirzaagha

luceos Thank you. Worked 😍

Mirzaagha

luceos but not understand. What is?

Elastic Endpoint
Username
Password
Index

and not have Analyzer language : Azerbaijan

and how to fixed size searchbox?

luceos

Mirzaagha Elastic Endpoint
Username
Password
Index

these are settings your elasticsearch instance has

Mirzaagha Azerbaijan

I don't know, your elastic instance probably needs to be configured with it. We currently have no flexible way to set this as a value, but you could modify the setting inside your database to the given analyzer, it might work. Otherwise a PR with the changes needed to make the field flexible would be appreciated.

Mirzaagha

🤔

fakruzaruret

may be orama tool is usable for text searches. I would like to share as an option
https://docs.askorama.ai/

rafaucau

@luceos I cannot get this extension work. After running the php flarum blomstra:search:index --recreate command, it seems to index everything ("Pushed a total of 5309387 into the index."), but the search results are empty. There are no errors.

I checked the indexes, and it looks like nothing is saved though, because the index takes up very little space and I tested it on 5 million posts.

~$ curl http://localhost:9200/_cat/indices?pretty=true
yellow open localhost 9cVCxB2gQ6W_ZTvCMOWjvg 1 1 0 0 249b 249b 249b

When I run the php flarum queue:work command, there are these errors in the logs:

[2024-06-22 09:44:03] flarum.ERROR: Blomstra\Search\Exceptions\SeedingException: Failed to seed: Index 0 out of bounds for length 0 in /var/www/html/vendor/blomstra/search/src/Jobs/SavingJob.php:53
Stack trace:
#0 /var/www/html/vendor/illuminate/container/BoundMethod.php(36): Blomstra\Search\Jobs\SavingJob->handle(Object(Elasticsearch\Client))
#1 /var/www/html/vendor/illuminate/container/Util.php(40): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}()
#2 /var/www/html/vendor/illuminate/container/BoundMethod.php(93): Illuminate\Container\Util::unwrapIfClosure(Object(Closure))
#3 /var/www/html/vendor/illuminate/container/BoundMethod.php(35): Illuminate\Container\BoundMethod::callBoundMethod(Object(Illuminate\Container\Container), Array, Object(Closure))
#4 /var/www/html/vendor/illuminate/container/Container.php(653): Illuminate\Container\BoundMethod::call(Object(Illuminate\Container\Container), Array, Array, NULL)
#5 /var/www/html/vendor/illuminate/bus/Dispatcher.php(128): Illuminate\Container\Container->call(Array)
#6 /var/www/html/vendor/illuminate/pipeline/Pipeline.php(128): Illuminate\Bus\Dispatcher->Illuminate\Bus\{closure}(Object(Blomstra\Search\Jobs\SavingJob))
#7 /var/www/html/vendor/illuminate/pipeline/Pipeline.php(103): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Blomstra\Search\Jobs\SavingJob))
#8 /var/www/html/vendor/illuminate/bus/Dispatcher.php(132): Illuminate\Pipeline\Pipeline->then(Object(Closure))
#9 /var/www/html/vendor/illuminate/queue/CallQueuedHandler.php(119): Illuminate\Bus\Dispatcher->dispatchNow(Object(Blomstra\Search\Jobs\SavingJob), false)
#10 /var/www/html/vendor/illuminate/pipeline/Pipeline.php(128): Illuminate\Queue\CallQueuedHandler->Illuminate\Queue\{closure}(Object(Blomstra\Search\Jobs\SavingJob))
#11 /var/www/html/vendor/illuminate/pipeline/Pipeline.php(103): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Blomstra\Search\Jobs\SavingJob))
#12 /var/www/html/vendor/illuminate/queue/CallQueuedHandler.php(118): Illuminate\Pipeline\Pipeline->then(Object(Closure))
#13 /var/www/html/vendor/illuminate/queue/CallQueuedHandler.php(70): Illuminate\Queue\CallQueuedHandler->dispatchThroughMiddleware(Object(Illuminate\Queue\Jobs\RedisJob), Object(Blomstra\Search\Jobs\SavingJob))
#14 /var/www/html/vendor/illuminate/queue/Jobs/Job.php(98): Illuminate\Queue\CallQueuedHandler->call(Object(Illuminate\Queue\Jobs\RedisJob), Array)
#15 /var/www/html/vendor/illuminate/queue/Worker.php(428): Illuminate\Queue\Jobs\Job->fire()
#16 /var/www/html/vendor/illuminate/queue/Worker.php(378): Illuminate\Queue\Worker->process(NULL, Object(Illuminate\Queue\Jobs\RedisJob), Object(Illuminate\Queue\WorkerOptions))
#17 /var/www/html/vendor/illuminate/queue/Worker.php(172): Illuminate\Queue\Worker->runJob(Object(Illuminate\Queue\Jobs\RedisJob), NULL, Object(Illuminate\Queue\WorkerOptions))
#18 /var/www/html/vendor/illuminate/queue/Console/WorkCommand.php(116): Illuminate\Queue\Worker->daemon(NULL, 'default', Object(Illuminate\Queue\WorkerOptions))
#19 /var/www/html/vendor/illuminate/queue/Console/WorkCommand.php(100): Illuminate\Queue\Console\WorkCommand->runWorker(NULL, 'default')
#20 /var/www/html/vendor/illuminate/container/BoundMethod.php(36): Illuminate\Queue\Console\WorkCommand->handle()
#21 /var/www/html/vendor/illuminate/container/Util.php(40): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}()
#22 /var/www/html/vendor/illuminate/container/BoundMethod.php(93): Illuminate\Container\Util::unwrapIfClosure(Object(Closure))
#23 /var/www/html/vendor/illuminate/container/BoundMethod.php(35): Illuminate\Container\BoundMethod::callBoundMethod(Object(Illuminate\Container\Container), Array, Object(Closure))
#24 /var/www/html/vendor/illuminate/container/Container.php(653): Illuminate\Container\BoundMethod::call(Object(Illuminate\Container\Container), Array, Array, NULL)
#25 /var/www/html/vendor/illuminate/console/Command.php(136): Illuminate\Container\Container->call(Array)
#26 /var/www/html/vendor/symfony/console/Command/Command.php(298): Illuminate\Console\Command->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Illuminate\Console\OutputStyle))
#27 /var/www/html/vendor/illuminate/console/Command.php(120): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Illuminate\Console\OutputStyle))
#28 /var/www/html/vendor/symfony/console/Application.php(1058): Illuminate\Console\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#29 /var/www/html/vendor/symfony/console/Application.php(301): Symfony\Component\Console\Application->doRunCommand(Object(Flarum\Queue\Console\WorkCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#30 /var/www/html/vendor/symfony/console/Application.php(171): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#31 /var/www/html/vendor/flarum/core/src/Console/Server.php(42): Symfony\Component\Console\Application->run()
#32 /var/www/html/flarum(24): Flarum\Console\Server->listen()
#33 {main}

Elasticsearch version: 8.13.4

luceos

rafaucau i will need to look into it. Can you share more information, like how is your queue set up? Output of php flarum info might be unnecessary, but still useful in case something is different than our usual set up.

rafaucau

luceos like how is your queue set up

I'm using blomstra/flarum-redis for queue.

Flarum info:
I'm testing this extension on a local docker environment currently.

Flarum core: 1.8.5
PHP version: 8.3.8
MySQL version: 10.5.23-MariaDB-1:10.5.23+maria~ubu2004
Loaded extensions: Core, date, libxml, openssl, pcre, sqlite3, zlib, ctype, curl, dom, fileinfo, filter, hash, iconv, json, mbstring, SPL, session, PDO, pdo_sqlite, standard, posix, random, readline, Reflection, Phar, SimpleXML, tokenizer, xml, xmlreader, xmlwriter, mysqlnd, gd, pdo_mysql, sodium, zip
+-----------------------------------+---------------------------------------+--------+
| Flarum Extensions                 |                                       |        |
+-----------------------------------+---------------------------------------+--------+
| ID                                | Version                               | Commit |
+-----------------------------------+---------------------------------------+--------+
| flarum-flags                      | v1.8.0                                |        |
| flarum-tags                       | v1.8.0                                |        |
| flarum-approval                   | v1.8.1                                |        |
| flarum-lock                       | v1.8.0                                |        |
| blomstra-gdpr                     | 0.1.0-beta.20                         |        |
| flarum-mentions                   | v1.8.3                                |        |
| flarum-subscriptions              | v1.8.0                                |        |
| v17development-support            | v2.7.2                                |        |
| flarum-suspend                    | v1.8.1                                |        |
| fof-follow-tags                   | 1.2.2                                 |        |
| fof-byobu                         | 1.3.6                                 |        |
| acpl-mobile-tab                   | 1.4.3                                 |        |
| fof-user-directory                | 1.3.3                                 |        |
| acpl-communities                  | dev-master                            |        |
| flarum-markdown                   | v1.8.0                                |        |
| fof-masquerade                    | 2.1.4                                 |        |
| fof-pages                         | 1.0.8                                 |        |
| fof-best-answer                   | 1.4.4                                 |        |
| fof-oauth                         | 1.6.9                                 |        |
| fof-drafts                        | 1.2.10                                |        |
| acpl-gamification                 | dev-master                            |        |
| fof-nightmode                     | 1.5.3                                 |        |
| flarumite-simple-discussion-views | 1.2.2                                 |        |
| v17development-seo                | v1.8.1                                |        |
| the-turk-diff                     | dev-fix-parse-and-render-missing-args |        |
| sycho-move-posts                  | v0.1.7                                |        |
| ianm-follow-users                 | 1.4.6                                 |        |
| fof-username-request              | 1.2.4                                 |        |
| fof-user-bio                      | 1.3.2                                 |        |
| fof-upload                        | 1.5.5                                 |        |
| fof-terms                         | 1.3.0                                 |        |
| fof-socialprofile                 | 1.1.6                                 |        |
| fof-polls                         | 2.2.2                                 |        |
| fof-merge-discussions             | 1.4.1                                 |        |
| fof-ignore-users                  | 1.2.1                                 |        |
| fof-geoip                         | 1.4.3                                 |        |
| fof-formatting                    | 1.0.3                                 |        |
| fof-anti-spam                     | 1.1.2                                 |        |
| flarum-statistics                 | v1.8.0                                |        |
| flarum-lang-polish                | dev-master                            |        |
| flarum-lang-english               | v1.8.0                                |        |
| flarum-emoji                      | v1.8.0                                |        |
| flarum-bbcode                     | v1.8.0                                |        |
| clarkwinkelmann-author-change     | 1.0.3                                 |        |
| askvortsov-rich-text              | v2.1.7                                |        |
| askvortsov-pwa                    | v3.4.1                                |        |
| askvortsov-moderator-warnings     | v0.6.3                                |        |
| acpl-moderation-tools             | dev-master                            |        |
| acpl-forum                        | dev-master                            |        |
| acpl-follow-communities           | dev-master                            |        |
| acpl-ads                          | dev-master                            |        |
+-----------------------------------+---------------------------------------+--------+
Base URL: http://facp.localhost
Installation path: /var/www/html
Queue driver: redis
Session driver: redis (Code override. Configured to file)
Mail driver: log
Debug mode: ON

There are custom extensions here, but I tested with them disabled also.

maisen20

Unfortunately, my posts/discussions have not been synchronized with the elasticsearch for weeks or months. Reindex works via manual command per cli. I have been searching for the problem for about 6 hours now, but apparently I don't quite understand it.

Please note the following:

Redis works without problems
Other jobs are processed without any issue by the system (e.g. Flarum\Mentions\Job\SendMentionsNotificationsJob)
Elasticsearch works
Manual recreation of the elasticsearch index works

The problem is on prod and dev environment.

Flarum core: 1.8.5
PHP version: 8.2.22
MySQL version: 10.6.19-MariaDB-ubu2004
Loaded extensions: Core, date, libxml, openssl, pcre, sqlite3, zlib, ctype, curl, dom, fileinfo, filter, hash, iconv, json, mbstring, SPL, session, PDO, pdo_sqlite, standard, posix, random, readline, Reflection, Phar, SimpleXML, tokenizer, xml, xmlreader, xmlwriter, mysqlnd, bcmath, calendar, exif, gd, gettext, gmp, imagick, imap, intl, memcached, mysqli, pcntl, pdo_mysql, redis, sodium, zip
+--------------------------------------+----------+--------+
| Flarum Extensions                    |          |        |
+--------------------------------------+----------+--------+
| ID                                   | Version  | Commit |
+--------------------------------------+----------+--------+
| flarum-flags                         | v1.8.0   |        |
| flarum-tags                          | v1.8.0   |        |
| flarum-approval                      | v1.8.1   |        |
| flarum-suspend                       | v1.8.1   |        |
| flarum-mentions                      | v1.8.3   |        |
| flarum-subscriptions                 | v1.8.0   |        |
| flarum-markdown                      | v1.8.0   |        |
| flarum-sticky                        | v1.8.0   |        |
| flamarkt-backoffice                  | 0.1.3    |        |
| fof-byobu                            | 1.3.6    |        |
| fof-user-directory                   | 1.3.3    |        |
| fof-follow-tags                      | 1.2.2    |        |
| therealsujitk-gifs                   | v4.1.1   |        |
| the-turk-stickiest                   | 3.0.1    |        |
| maicol07-sso                         | 1.11.5   |        |
| ianm-synopsis                        | 1.3.2    |        |
| fof-user-bio                         | 1.3.2    |        |
| fof-upload                           | 1.5.4    |        |
| fof-subscribed                       | 1.1.4    |        |
| fof-links                            | 1.2.3    |        |
| fof-linguist                         | 1.1.2    |        |
| fof-impersonate                      | 1.1.1    |        |
| fof-ignore-users                     | 1.2.1    |        |
| fof-drafts                           | 1.2.10   |        |
| fof-default-user-preferences         | 1.2.1    |        |
| fof-best-answer                      | 1.4.2    |        |
| flarum-statistics                    | v1.8.0   |        |
| flarum-nicknames                     | v1.8.0   |        |
| flarum-lock                          | v1.8.0   |        |
| flarum-likes                         | v1.8.0   |        |
| flarum-lang-german                   | 1.10.0   |        |
| flarum-emoji                         | v1.8.0   |        |
| flarum-bbcode                        | v1.8.0   |        |
| flamarkt-matomo                      | 0.1.0    |        |
| datlechin-link-preview               | v1.4.0   |        |
| datlechin-landing-page               | 0.1.0    |        |
| clarkwinkelmann-post-stream-search   | 1.1.0    |        |
| clarkwinkelmann-first-post-approval  | 1.0.1    |        |
| clarkwinkelmann-discussion-bookmarks | 2.0.1    |        |
| clarkwinkelmann-collapsible-posts    | 1.0.1    |        |
| blomstra-search                      | 0.1.19   |        |
| askvortsov-rich-text                 | v2.1.7   |        |
| askvortsov-moderator-warnings        | v0.6.3   |        |
| askvortsov-markdown-tables           | v1.2.1   |        |
+--------------------------------------+----------+--------+
Base URL: ***
Installation path: ***
Queue driver: redis
Session driver: redis (Code override. Configured to file)
Scheduler status: aktiv
Mail driver: smtp
Debug mode: off

« Previous Page Next Page »