• Extensions
  • Blomstra Search, an elasticsearch fulltext search extension

To inform everyone:

I figured out what the reason for the limitation to 10 characters is.
In the file "/vendor/blomstra/search/src/Commands/BuildCommand.php" there are the values for 'index.max_ngram_diff' and 'max_gram' which tell ES's tokenizer how long the token can be. These values are set to 10. I assume that this is the case for all of you.

This means words containing more than 10 characters will not be tokenized and therefore these words/posts will not be found.

5 months later

composer require blomstra/search:"*"

l want install this extension but write a error words

    luceos but not understand. What is?

    Elastic Endpoint
    Username
    Password
    Index

    and not have Analyzer language : Azerbaijan

    and how to fixed size searchbox?

      Mirzaagha Elastic Endpoint
      Username
      Password
      Index

      these are settings your elasticsearch instance has

      Mirzaagha Azerbaijan

      I don't know, your elastic instance probably needs to be configured with it. We currently have no flexible way to set this as a value, but you could modify the setting inside your database to the given analyzer, it might work. Otherwise a PR with the changes needed to make the field flexible would be appreciated.

      2 months later
      2 months later

      @luceos I cannot get this extension work. After running the php flarum blomstra:search:index --recreate command, it seems to index everything ("Pushed a total of 5309387 into the index."), but the search results are empty. There are no errors.

      I checked the indexes, and it looks like nothing is saved though, because the index takes up very little space and I tested it on 5 million posts.

      ~$ curl http://localhost:9200/_cat/indices?pretty=true
      yellow open localhost 9cVCxB2gQ6W_ZTvCMOWjvg 1 1 0 0 249b 249b 249b

      When I run the php flarum queue:work command, there are these errors in the logs:

      [2024-06-22 09:44:03] flarum.ERROR: Blomstra\Search\Exceptions\SeedingException: Failed to seed: Index 0 out of bounds for length 0 in /var/www/html/vendor/blomstra/search/src/Jobs/SavingJob.php:53
      Stack trace:
      #0 /var/www/html/vendor/illuminate/container/BoundMethod.php(36): Blomstra\Search\Jobs\SavingJob->handle(Object(Elasticsearch\Client))
      #1 /var/www/html/vendor/illuminate/container/Util.php(40): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}()
      #2 /var/www/html/vendor/illuminate/container/BoundMethod.php(93): Illuminate\Container\Util::unwrapIfClosure(Object(Closure))
      #3 /var/www/html/vendor/illuminate/container/BoundMethod.php(35): Illuminate\Container\BoundMethod::callBoundMethod(Object(Illuminate\Container\Container), Array, Object(Closure))
      #4 /var/www/html/vendor/illuminate/container/Container.php(653): Illuminate\Container\BoundMethod::call(Object(Illuminate\Container\Container), Array, Array, NULL)
      #5 /var/www/html/vendor/illuminate/bus/Dispatcher.php(128): Illuminate\Container\Container->call(Array)
      #6 /var/www/html/vendor/illuminate/pipeline/Pipeline.php(128): Illuminate\Bus\Dispatcher->Illuminate\Bus\{closure}(Object(Blomstra\Search\Jobs\SavingJob))
      #7 /var/www/html/vendor/illuminate/pipeline/Pipeline.php(103): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Blomstra\Search\Jobs\SavingJob))
      #8 /var/www/html/vendor/illuminate/bus/Dispatcher.php(132): Illuminate\Pipeline\Pipeline->then(Object(Closure))
      #9 /var/www/html/vendor/illuminate/queue/CallQueuedHandler.php(119): Illuminate\Bus\Dispatcher->dispatchNow(Object(Blomstra\Search\Jobs\SavingJob), false)
      #10 /var/www/html/vendor/illuminate/pipeline/Pipeline.php(128): Illuminate\Queue\CallQueuedHandler->Illuminate\Queue\{closure}(Object(Blomstra\Search\Jobs\SavingJob))
      #11 /var/www/html/vendor/illuminate/pipeline/Pipeline.php(103): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Blomstra\Search\Jobs\SavingJob))
      #12 /var/www/html/vendor/illuminate/queue/CallQueuedHandler.php(118): Illuminate\Pipeline\Pipeline->then(Object(Closure))
      #13 /var/www/html/vendor/illuminate/queue/CallQueuedHandler.php(70): Illuminate\Queue\CallQueuedHandler->dispatchThroughMiddleware(Object(Illuminate\Queue\Jobs\RedisJob), Object(Blomstra\Search\Jobs\SavingJob))
      #14 /var/www/html/vendor/illuminate/queue/Jobs/Job.php(98): Illuminate\Queue\CallQueuedHandler->call(Object(Illuminate\Queue\Jobs\RedisJob), Array)
      #15 /var/www/html/vendor/illuminate/queue/Worker.php(428): Illuminate\Queue\Jobs\Job->fire()
      #16 /var/www/html/vendor/illuminate/queue/Worker.php(378): Illuminate\Queue\Worker->process(NULL, Object(Illuminate\Queue\Jobs\RedisJob), Object(Illuminate\Queue\WorkerOptions))
      #17 /var/www/html/vendor/illuminate/queue/Worker.php(172): Illuminate\Queue\Worker->runJob(Object(Illuminate\Queue\Jobs\RedisJob), NULL, Object(Illuminate\Queue\WorkerOptions))
      #18 /var/www/html/vendor/illuminate/queue/Console/WorkCommand.php(116): Illuminate\Queue\Worker->daemon(NULL, 'default', Object(Illuminate\Queue\WorkerOptions))
      #19 /var/www/html/vendor/illuminate/queue/Console/WorkCommand.php(100): Illuminate\Queue\Console\WorkCommand->runWorker(NULL, 'default')
      #20 /var/www/html/vendor/illuminate/container/BoundMethod.php(36): Illuminate\Queue\Console\WorkCommand->handle()
      #21 /var/www/html/vendor/illuminate/container/Util.php(40): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}()
      #22 /var/www/html/vendor/illuminate/container/BoundMethod.php(93): Illuminate\Container\Util::unwrapIfClosure(Object(Closure))
      #23 /var/www/html/vendor/illuminate/container/BoundMethod.php(35): Illuminate\Container\BoundMethod::callBoundMethod(Object(Illuminate\Container\Container), Array, Object(Closure))
      #24 /var/www/html/vendor/illuminate/container/Container.php(653): Illuminate\Container\BoundMethod::call(Object(Illuminate\Container\Container), Array, Array, NULL)
      #25 /var/www/html/vendor/illuminate/console/Command.php(136): Illuminate\Container\Container->call(Array)
      #26 /var/www/html/vendor/symfony/console/Command/Command.php(298): Illuminate\Console\Command->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Illuminate\Console\OutputStyle))
      #27 /var/www/html/vendor/illuminate/console/Command.php(120): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Illuminate\Console\OutputStyle))
      #28 /var/www/html/vendor/symfony/console/Application.php(1058): Illuminate\Console\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
      #29 /var/www/html/vendor/symfony/console/Application.php(301): Symfony\Component\Console\Application->doRunCommand(Object(Flarum\Queue\Console\WorkCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
      #30 /var/www/html/vendor/symfony/console/Application.php(171): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
      #31 /var/www/html/vendor/flarum/core/src/Console/Server.php(42): Symfony\Component\Console\Application->run()
      #32 /var/www/html/flarum(24): Flarum\Console\Server->listen()
      #33 {main}  

      Elasticsearch version: 8.13.4

        rafaucau i will need to look into it. Can you share more information, like how is your queue set up? Output of php flarum info might be unnecessary, but still useful in case something is different than our usual set up.

          luceos like how is your queue set up

          I'm using blomstra/flarum-redis for queue.

          Flarum info:
          I'm testing this extension on a local docker environment currently.

          Flarum core: 1.8.5
          PHP version: 8.3.8
          MySQL version: 10.5.23-MariaDB-1:10.5.23+maria~ubu2004
          Loaded extensions: Core, date, libxml, openssl, pcre, sqlite3, zlib, ctype, curl, dom, fileinfo, filter, hash, iconv, json, mbstring, SPL, session, PDO, pdo_sqlite, standard, posix, random, readline, Reflection, Phar, SimpleXML, tokenizer, xml, xmlreader, xmlwriter, mysqlnd, gd, pdo_mysql, sodium, zip
          +-----------------------------------+---------------------------------------+--------+
          | Flarum Extensions                 |                                       |        |
          +-----------------------------------+---------------------------------------+--------+
          | ID                                | Version                               | Commit |
          +-----------------------------------+---------------------------------------+--------+
          | flarum-flags                      | v1.8.0                                |        |
          | flarum-tags                       | v1.8.0                                |        |
          | flarum-approval                   | v1.8.1                                |        |
          | flarum-lock                       | v1.8.0                                |        |
          | blomstra-gdpr                     | 0.1.0-beta.20                         |        |
          | flarum-mentions                   | v1.8.3                                |        |
          | flarum-subscriptions              | v1.8.0                                |        |
          | v17development-support            | v2.7.2                                |        |
          | flarum-suspend                    | v1.8.1                                |        |
          | fof-follow-tags                   | 1.2.2                                 |        |
          | fof-byobu                         | 1.3.6                                 |        |
          | acpl-mobile-tab                   | 1.4.3                                 |        |
          | fof-user-directory                | 1.3.3                                 |        |
          | acpl-communities                  | dev-master                            |        |
          | flarum-markdown                   | v1.8.0                                |        |
          | fof-masquerade                    | 2.1.4                                 |        |
          | fof-pages                         | 1.0.8                                 |        |
          | fof-best-answer                   | 1.4.4                                 |        |
          | fof-oauth                         | 1.6.9                                 |        |
          | fof-drafts                        | 1.2.10                                |        |
          | acpl-gamification                 | dev-master                            |        |
          | fof-nightmode                     | 1.5.3                                 |        |
          | flarumite-simple-discussion-views | 1.2.2                                 |        |
          | v17development-seo                | v1.8.1                                |        |
          | the-turk-diff                     | dev-fix-parse-and-render-missing-args |        |
          | sycho-move-posts                  | v0.1.7                                |        |
          | ianm-follow-users                 | 1.4.6                                 |        |
          | fof-username-request              | 1.2.4                                 |        |
          | fof-user-bio                      | 1.3.2                                 |        |
          | fof-upload                        | 1.5.5                                 |        |
          | fof-terms                         | 1.3.0                                 |        |
          | fof-socialprofile                 | 1.1.6                                 |        |
          | fof-polls                         | 2.2.2                                 |        |
          | fof-merge-discussions             | 1.4.1                                 |        |
          | fof-ignore-users                  | 1.2.1                                 |        |
          | fof-geoip                         | 1.4.3                                 |        |
          | fof-formatting                    | 1.0.3                                 |        |
          | fof-anti-spam                     | 1.1.2                                 |        |
          | flarum-statistics                 | v1.8.0                                |        |
          | flarum-lang-polish                | dev-master                            |        |
          | flarum-lang-english               | v1.8.0                                |        |
          | flarum-emoji                      | v1.8.0                                |        |
          | flarum-bbcode                     | v1.8.0                                |        |
          | clarkwinkelmann-author-change     | 1.0.3                                 |        |
          | askvortsov-rich-text              | v2.1.7                                |        |
          | askvortsov-pwa                    | v3.4.1                                |        |
          | askvortsov-moderator-warnings     | v0.6.3                                |        |
          | acpl-moderation-tools             | dev-master                            |        |
          | acpl-forum                        | dev-master                            |        |
          | acpl-follow-communities           | dev-master                            |        |
          | acpl-ads                          | dev-master                            |        |
          +-----------------------------------+---------------------------------------+--------+
          Base URL: http://facp.localhost
          Installation path: /var/www/html
          Queue driver: redis
          Session driver: redis (Code override. Configured to file)
          Mail driver: log
          Debug mode: ON

          There are custom extensions here, but I tested with them disabled also.

          2 months later

          Unfortunately, my posts/discussions have not been synchronized with the elasticsearch for weeks or months. Reindex works via manual command per cli. I have been searching for the problem for about 6 hours now, but apparently I don't quite understand it.

          Please note the following:

          • Redis works without problems
          • Other jobs are processed without any issue by the system (e.g. Flarum\Mentions\Job\SendMentionsNotificationsJob)
          • Elasticsearch works
          • Manual recreation of the elasticsearch index works

          The problem is on prod and dev environment.

          Flarum core: 1.8.5
          PHP version: 8.2.22
          MySQL version: 10.6.19-MariaDB-ubu2004
          Loaded extensions: Core, date, libxml, openssl, pcre, sqlite3, zlib, ctype, curl, dom, fileinfo, filter, hash, iconv, json, mbstring, SPL, session, PDO, pdo_sqlite, standard, posix, random, readline, Reflection, Phar, SimpleXML, tokenizer, xml, xmlreader, xmlwriter, mysqlnd, bcmath, calendar, exif, gd, gettext, gmp, imagick, imap, intl, memcached, mysqli, pcntl, pdo_mysql, redis, sodium, zip
          +--------------------------------------+----------+--------+
          | Flarum Extensions                    |          |        |
          +--------------------------------------+----------+--------+
          | ID                                   | Version  | Commit |
          +--------------------------------------+----------+--------+
          | flarum-flags                         | v1.8.0   |        |
          | flarum-tags                          | v1.8.0   |        |
          | flarum-approval                      | v1.8.1   |        |
          | flarum-suspend                       | v1.8.1   |        |
          | flarum-mentions                      | v1.8.3   |        |
          | flarum-subscriptions                 | v1.8.0   |        |
          | flarum-markdown                      | v1.8.0   |        |
          | flarum-sticky                        | v1.8.0   |        |
          | flamarkt-backoffice                  | 0.1.3    |        |
          | fof-byobu                            | 1.3.6    |        |
          | fof-user-directory                   | 1.3.3    |        |
          | fof-follow-tags                      | 1.2.2    |        |
          | therealsujitk-gifs                   | v4.1.1   |        |
          | the-turk-stickiest                   | 3.0.1    |        |
          | maicol07-sso                         | 1.11.5   |        |
          | ianm-synopsis                        | 1.3.2    |        |
          | fof-user-bio                         | 1.3.2    |        |
          | fof-upload                           | 1.5.4    |        |
          | fof-subscribed                       | 1.1.4    |        |
          | fof-links                            | 1.2.3    |        |
          | fof-linguist                         | 1.1.2    |        |
          | fof-impersonate                      | 1.1.1    |        |
          | fof-ignore-users                     | 1.2.1    |        |
          | fof-drafts                           | 1.2.10   |        |
          | fof-default-user-preferences         | 1.2.1    |        |
          | fof-best-answer                      | 1.4.2    |        |
          | flarum-statistics                    | v1.8.0   |        |
          | flarum-nicknames                     | v1.8.0   |        |
          | flarum-lock                          | v1.8.0   |        |
          | flarum-likes                         | v1.8.0   |        |
          | flarum-lang-german                   | 1.10.0   |        |
          | flarum-emoji                         | v1.8.0   |        |
          | flarum-bbcode                        | v1.8.0   |        |
          | flamarkt-matomo                      | 0.1.0    |        |
          | datlechin-link-preview               | v1.4.0   |        |
          | datlechin-landing-page               | 0.1.0    |        |
          | clarkwinkelmann-post-stream-search   | 1.1.0    |        |
          | clarkwinkelmann-first-post-approval  | 1.0.1    |        |
          | clarkwinkelmann-discussion-bookmarks | 2.0.1    |        |
          | clarkwinkelmann-collapsible-posts    | 1.0.1    |        |
          | blomstra-search                      | 0.1.19   |        |
          | askvortsov-rich-text                 | v2.1.7   |        |
          | askvortsov-moderator-warnings        | v0.6.3   |        |
          | askvortsov-markdown-tables           | v1.2.1   |        |
          +--------------------------------------+----------+--------+
          Base URL: ***
          Installation path: ***
          Queue driver: redis
          Session driver: redis (Code override. Configured to file)
          Scheduler status: aktiv
          Mail driver: smtp
          Debug mode: off
          11 days later

          As far as I can tell, changes to posts are not being updated in the Elasticsearch database. Or am I mistaken?

          a month later

          I can't get past the "No alive nodes in your cluser" message. Elasticsearch is running, there is a node, there is an index. I can get the list of indexes from it by calling curl blah blah /_aliases. I have entered the host/user/password/index correctly in the flarum admin, but I still get "No alive nodes in your cluster" when I try running php flarum blomstra:search:index

          I wonder ... does it have something to do with SSL? I noticed most examples above are http://localhost:9200. My elasticsearch seems to be running on HTTPS. Is that a problem?

            Shock no, we also use https in production. Just make sure your Elastic Endpoint setting uses that https url.

              luceos It does.

              When I query the service using the command from the ES docs, it works:

              curl --cacert $ES_HOME/config/certs/http_ca.crt -u elastic:$ELASTIC_PASSWORD https://localhost:9200
              yields
              {
              "name" : "node-1",
              "cluster_name" : "cluster-1",
              "cluster_uuid" : "jwMl-ycHSCiPyEwQKnAlyw",
              "version" : {
              "number" : "8.15.2",
              "build_flavor" : "default",
              "build_type" : "tar",
              "build_hash" : "98adf7bf6bb69b66ab95b761c9e5aadb0bb059a3",
              "build_date" : "2024-09-19T10:06:03.564235954Z",
              "build_snapshot" : false,
              "lucene_version" : "9.11.1",
              "minimum_wire_compatibility_version" : "7.17.0",
              "minimum_index_compatibility_version" : "7.0.0"
              },
              "tagline" : "You Know, for Search"
              }

              I have all the same settings set in flarum:

              host: https://localhost:9200
              username: elastic
              password: <$ELASTIC_PASSWORD>
              index: flarum-search

              and yet when I run php flarum blomstra:search:index i get

              In StaticNoPingConnectionPool.php line 64:
                                                      
                No alive nodes found in your cluster  

                Shock then the server flarum runs as probably can't connect to the redis server. Is this dockerized?

                  Shock sorry I meant elastic 🙈

                  still it seems it cannot connect to the elastic instance at all

                    17 days later