Scout Search

clarkwinkelmann

ernestdefoe sorry, haven't found time yet. Might take a few more weeks.

tohsakarat

Hello,
I tried to optimize Chinese search results by using a custom tokenizer for TNTSearch. However, I encountered an issue where the tokenizer does not seem to be working. Even though I set the tokenizer in flarumEgineManager.php as shown in the code below, it is not being called (I threw an error within the tokenizer, but the search service continued to run without any issues).
Do you have any insights or suggestions on what might be going wrong?

    {
        if (!class_exists(TNTSearch::class) || !class_exists(TNTSearchEngine::class)) {
            throw new Exception('Please install the TNTSearch scout package: teamtnt/laravel-scout-tntsearch-driver.');
        }

        $storage = resolve(Paths::class)->storage . '/tntsearch';

        // TNTSearch won't create the folder automatically, so we need to do it
        if (!file_exists($storage)) {
            mkdir($storage);
        }

        /**
         * @var SettingsRepositoryInterface $settings
         */
        $settings = resolve(SettingsRepositoryInterface::class);

        $tnt = new TNTSearch();
        $tokenizer= new TokenizerHandler();
        $tnt->loadConfig([
            'storage' => $storage,
            'searchBoolean' => true,
            'fuzziness' =>  (bool)$settings->get('clarkwinkelmann-scout.tntsearchFuzziness'),
            'fuzzy_distance '=>  $settings->get('clarkwinkelmann-scout.tntsearchFuzzyDistance') ?: 2,
            'fuzzy_prefix_length'=> $settings->get('clarkwinkelmann-scout.tntsearchFuzzyPrefixLength') ?: 50,
            'fuzzy_max_expansions '=> $settings->get('clarkwinkelmann-scout.tntsearchFuzzyMaxExpansions') ?: 2,
            'asYouType' => true,
            'maxDocs' => $settings->get('clarkwinkelmann-scout.tntsearchMaxDocs') ?: 500,
            'tokenizer' =>$tokenizer,
            'engine' => (\TeamTNT\TNTSearch\Engines\SqliteEngine::class)
            ]);
 #throw new Exception("123415142");        
      //  $tnt->setTokenizer($tokenizer);
        $tnt->setDatabaseHandle(resolve(ConnectionInterface::class)->getPdo());


        return new TNTSearchEngine($tnt);
    }

Thank you for your help!

clarkwinkelmann

tohsakarat is the driver set to tntsearch? If an exception doesn't throw, it's probably not loading at all.

Maybe an exception at this stage will cause an extension boot error. You could also try writing to the log file or die() instead to check if it works. It will only be instantiated during search or when a change need syncing to the driver (possibly in an async queue). Just opening the forum won't automatically trigger that code.

tohsakarat

clarkwinkelmann
Hello,
I have confirmed that Scout search with the TNT engine is working. Using a rather rudimentary approach to test step by step, I discovered that when calling the search API, the constructor of the TeamTNT\TNTSearch class is triggered. However, the core search method is not being called.
I am wondering if this method might have been overridden by some intermediary layer.
Do you have any insights or suggestions on what might be causing this issue?
Thank you for your help!

clarkwinkelmann

tohsakarat can you give me the exact import paths to the classes/methods you are referencing? I'm not sure what you mean by core search. This extension overrides the original full text search gambits of Flarum clarkwinkelmann/flarum-ext-scoutblob/4e1eecd18f687b3105c8dd3cc6ca5f8dc434729d/extend.php#L28-L31

tohsakarat

clarkwinkelmann
l

Hello,

My experiments are based on modifying the /src/FlarumEngineManager.php file in your project. Here is the link to the file: FlarumEngineManager.php (I know it's not a good development practice to tamper with the project like this, lol, but this is just an experiment).

I have midfield the public function createTntsearchDriver(): TNTSearchEngine method and introduced a custom tokenizer within it. However, I found that this tokenizer does not seem to be working. So, I traced the issue all the way to vendor/teamtnt/tntsearch/src/TNTSearch.php. Here is the link to the file: TNTSearch.php, and discovered that the search function does not seem to be working. I guess it is overwritten.

Thank you for your help!

tohsakarat

clarkwinkelmann

Hello!

I believe I have identified the root cause of the issue we discussed. It appears to be related to line 76 of the FlarumEngineManager.php file in your repository:

clarkwinkelmann/flarum-ext-scoutblob/4e1eecd18f687b3105c8dd3cc6ca5f8dc434729d/src/FlarumEngineManager.php

The line sets the searchBoolean parameter to true, and as you may know, TNTSearch does not utilize a tokenizer when searchBoolean is enabled.

I would like to know if it's possible to disable the Boolean search feature. I conducted some preliminary tests with this feature turned off, and it seems to work fine. However, I am uncertain if disabling Boolean search might affect other scenarios or functionalities.

Could you please provide some guidance on whether it's safe to disable Boolean search, or if there are any potential side effects I should be aware of?

Additionally, since TNTSearch is a PHP implementation, it is suitable for use on shared hosting environments. Plus many users may not have the expertise to configure other search services, making TNTSearch quite indispensable for them.

Thank you for your help!!!

clarkwinkelmann

tohsakarat as I shared in earlier posts, despite all my tests with TNT, I wasn't able to get half of their advertised features to work, so maybe I have something misconfigured in the extension itself. I must have enabled boolean search based on the description in their documentation, but I didn't remember what exactly it provided.

It should be fine to change any of the settings, it's just full text search after all, none of the Scout drivers need to handle any special input other than the text provided by the user.

llaumgui

OK. Very good extension with Meilisearch !
Is there a way to make a better ranking in the discusion title ?

llaumgui

Hi, Since few day and update:

2024-09-11 07:24:16] flarum.ERROR: Illuminate\Contracts\Container\BindingResolutionException: Unresolvable dependency resolving [Parameter #0 [ <required> string $url ]] in class Meilisearch\Client in /vendor/illuminate/container/Container.php:1104

clarkwinkelmann

llaumgui I'll try to take a look over the next few days.

In the meantime, you should maybe try to force a specific Meilisearch PHP client version using the command in the README.

Edit: I see the same problem was discussed in this thread back in January and fixed itself, so maybe it's related to another extension being used at the same time.

llaumgui

clarkwinkelmann In the meantime, you should maybe try to force a specific Meilisearch PHP client version using the command in the README.

Yes, I switch to 0.23 and it's work. What is the last version working ?

llaumgui

Question; with meilisearch, is there a way to boost recent post ?
I thinks index is only build with id / content without date ?

clarkwinkelmann

llaumgui that's correct, dates are not indexed are the moment. Even if they were, I'm not sure how we could use this in the Meilisearch query?

llaumgui

clarkwinkelmann not in query, in index configuration. To use ranking: https://www.meilisearch.com/docs/learn/relevancy/custom_ranking_rules#:~:text=Meilisearch%20supports%20two%20types%20of,or%20desc%20for%20descending%20order.

Teddan

I have everything working just fine with Meilisearch. Thanks for the extensions.

I wonder if it's possible to differentiate the searching result by Discussion(title) or Posts (content) just as what ↓ does?

and P.S. Could we target a post by it's post ID (not discussion ID)?

clarkwinkelmann

Teddan I don't understand your question. The posts and discussions indexes are combined in the results to offer the Flarum full text search. There is currently no way to change the weight of each index in the algorithm. There is also no gambits in place to search specific attributes, but this could be implemented and use Scout as well.

You can search in posts with my post-stream-search extension. It will use the posts index to search within a discussion.

stl

Hi , I installed the extension and install teamtnt/laravel-scout-tntsearch-driver as well.
When I run command php flarum scout:import-all , I got these errors.
And TNTSearch is not working.

[29-Jan-2025 07:16:57 UTC] PHP Deprecated:  Creation of dynamic property TeamTNT\TNTSearch\TNTSearch::$fuzziness is deprecated in /home/user1/public_html/flarum/vendor/clarkwinkelmann/flarum-ext-scout/src/FlarumEngineManager.php on line 80
[29-Jan-2025 07:16:57 UTC] PHP Deprecated:  Creation of dynamic property TeamTNT\TNTSearch\TNTSearch::$fuzzy_distance is deprecated in /home/user1/public_html/flarum/vendor/clarkwinkelmann/flarum-ext-scout/src/FlarumEngineManager.php on line 83
[29-Jan-2025 07:16:57 UTC] PHP Deprecated:  Creation of dynamic property TeamTNT\TNTSearch\TNTSearch::$fuzzy_prefix_length is deprecated in /home/user1/public_html/flarum/vendor/clarkwinkelmann/flarum-ext-scout/src/FlarumEngineManager.php on line 84
[29-Jan-2025 07:16:57 UTC] PHP Deprecated:  Creation of dynamic property TeamTNT\TNTSearch\TNTSearch::$fuzzy_max_expansions is deprecated in /home/user1/public_html/flarum/vendor/clarkwinkelmann/flarum-ext-scout/src/FlarumEngineManager.php on line 85
[29-Jan-2025 07:16:57 UTC] PHP Deprecated:  Creation of dynamic property TeamTNT\TNTSearch\TNTSearch::$asYouType is deprecated in /home/user1/public_html/flarum/vendor/clarkwinkelmann/flarum-ext-scout/src/FlarumEngineManager.php on line 86
[29-Jan-2025 07:16:59 UTC] PHP Deprecated:  Optional parameter $upload declared before required parameter $contents is implicitly treated as a required parameter in /home/user1/public_html/flarum/vendor/fof/upload/src/Adapters/Flysystem.php on line 55
[29-Jan-2025 07:16:59 UTC] PHP Deprecated:  mb_strtolower(): Passing null to parameter #1 ($string) of type string is deprecated in /home/user1/public_html/flarum/vendor/teamtnt/tntsearch/src/Support/Tokenizer.php on line 10

Here is my Flarum info

Flarum core: 1.8.9
PHP version: 8.3.14
MySQL version: 10.5.27-MariaDB-cll-lve
Loaded extensions: Core, date, libxml, openssl, pcre, zlib, filter, hash, json, pcntl, random, readline, Reflection, SPL, session, standard, bcmath, bz2, calendar, ctype, curl, dba, dom, enchant, mbstring, fileinfo, ftp, gd, gettext, gmp, iconv,imap, intl, ldap, exif, mysqlnd, odbc, PDO, pgsql, Phar, posix, pspell, shmop, SimpleXML, snmp, soap, sockets, sqlite3, sysvmsg, sysvsem, sysvshm, tidy, tokenizer, xml, xmlwriter, xsl, zip, mysqli, pdo_mysql, PDO_ODBC, pdo_pgsql, pdo_sqlite, xmlreader, i360, Zend OPcache
+-------------------------------------------+---------+--------+
| Flarum Extensions                         |         |        |
+-------------------------------------------+---------+--------+
| ID                                        | Version | Commit |
+-------------------------------------------+---------+--------+
| flarum-flags                              | v1.8.2  |        |
| flarum-sticky                             | v1.8.2  |        |
| flarum-tags                               | v1.8.3  |        |
| flarum-lock                               | v1.8.2  |        |
| flarum-approval                           | v1.8.2  |        |
| clarkwinkelmann-advanced-search-highlight | 1.0.3   |        |
| v17development-blog                       | v0.8.0  |        |
| pattinsonfuture-lang-traditional-chinese  | 1.0.2   |        |
| kk14569-moderno                           | v1.0.3  |        |
| fof-upload                                | 1.7.1   |        |
| fof-formatting                            | 1.0.3   |        |
| flarum-suspend                            | v1.8.4  |        |
| flarum-subscriptions                      | v1.8.1  |        |
| flarum-statistics                         | v1.8.1  |        |
| flarum-mentions                           | v1.8.5  |        |
| flarum-markdown                           | v1.8.1  |        |
| flarum-likes                              | v1.8.1  |        |
| flarum-lang-english                       | v1.8.0  |        |
| flarum-extension-manager                  | v1.0.6  |        |
| flarum-emoji                              | v1.8.1  |        |
| flarum-bbcode                             | v1.8.0  |        |
| clarkwinkelmann-scout                     | 0.3.5   |        |
| askvortsov-markdown-tables                | v1.2.1  |        |
+-------------------------------------------+---------+--------+

Can anyone tell me what to do now ?

clarkwinkelmann

stl what makes you think it doesn't work?

If the deprecation warnings only appear over the command line, they shouldn't prevent anything from working. You should in any case disable deprecation warnings in your php.ini (and set display_errors=Off) to prevent warnings from breaking the Flarum API.

I couldn't fix those warnings even if I wanted to as they are in TNT source code, not mine, so nothing to do except ignore them.

stl

Thanks for your reply.
I thought these messages mean that something went wrong.

I try to search , but there is no result.
When I switch to meilisearch , I can get search results.
but no result if I switch to TNTSearch.

That's why I think it dose not work.
Do I need to configure something or install any other packages?

clarkwinkelmann

stl try disabling the PHP warning output as suggested in my previous post then try re-indexing. If it still doesn't work I have no idea what the problem could be.

As discussed earlier in this thread, when I implemented TNT support it worked but was basically no better than Flarum's built-in search. I couldn't get any of the advertised features of TNT to work except basic search. For those reasons I recommend not bothering with TNT at all.

« Previous Page