Blomstra Search, an elasticsearch fulltext search extension

luceos

Teddan can you share the output of php flarum info and the elastic server version?

Teddan

This is my info:

Flarum core 1.3.0
PHP version: 8.0.17
MySQL version: 5.7.37-log
Loaded extensions: Core, date, libxml, openssl, pcre, sqlite3, zlib, bcmath, ctype, curl, dom, filter, ftp, gd, gettext, hash, iconv, intl, json, mbstring, SPL, session, pcntl, standard, PDO, mysqlnd, pdo_sqlite, Phar, posix, Reflection, pdo_mysql, shmop, SimpleXML, soap, sockets, sodium, mysqli, sysvsem, tokenizer, xml, xmlreader, xmlwriter, fileinfo, exif, gmp, imagick, Zend OPcache
+------------------------------------------+-------------+--------+
| Flarum Extensions | | |
+------------------------------------------+-------------+--------+
| ID | Version | Commit |
+------------------------------------------+-------------+--------+
| flarum-flags | v1.3.0 | |
| flarum-suspend | v1.3.0 | |
| flarum-subscriptions | v1.3.0 | |
| flarum-tags | v1.3.0 | |
| flarum-approval | v1.3.0 | |
| flarum-bbcode | v1.3.0 | |
| fof-upload | 1.2.1 | |
| askvortsov-auto-moderator | v0.1.2 | |
| flarum-sticky | v1.3.0 | |
| fof-user-directory | 1.2.2 | |
| fof-follow-tags | 1.1.3 | |
| flarum-nicknames | v1.2.0 | |
| fof-byobu | 1.1.6 | |
| flarumite-simple-discussion-views | 1.1.3 | |
| afrux-forum-widgets-core | v0.1.7 | |
| zerosonesfun-member | 1.2 | |
| zerosonesfun-bbcode-iframe | 1.0 | |
| zerosonesfun-bbcode-button | 1.1.2 | |
| zerosonesfun-bbcode-audio | 3.1.2 | |
| zequeen-acgembed-remastered | v1.1.2 | |
| yannisme-oxotheme | 2.1.3 | |
| v17development-user-badges | v1.0.0 | |
| v17development-seo | v1.8.0 | |
| the-turk-stickiest | 2.0.3 | |
| the-turk-flamoji | 1.0.3 | |
| the-turk-diff | 1.1.1 | |
| sycho-profile-cover | v1.3.1 | |
| sycho-private-facade | v0.1.6 | |
| sycho-advanced-extension-categories | v0.1.3 | |
| swaggymacro-only-starter | 0.6.2 | |
| nearata-signup-confirm-password | v3.0.0 | |
| nearata-internal-links-noreload | v2.0.1 | |
| nearata-cakeday | v2.0.0 | |
| malago-achievements | 0.5.4 | |
| littlecxm-reply-to-see | 1.0.2 | |
| kilowhat-rich-embeds | 1.0.2 | |
| justoverclock-hashtag | 0.2.3 | |
| justoverclock-feedback | 0.1.9 | |
| justoverclock-auto-post-badge-pro | 0.2.1 | |
| itnt-uitab | v0.4.2 | |
| ianm-synopsis | 1.2.0 | |
| ianm-html-head | 1.1.0 | |
| ianm-follow-users | 1.0.5 | |
| fof-username-request | 1.0.1 | |
| fof-user-bio | 1.0.2 | |
| fof-terms | 1.1.2 | |
| fof-profile-image-crop | 1.0.1 | |
| fof-prevent-necrobumping | 1.0.1 | |
| fof-pretty-mail | 1.0.1 | |
| fof-polls | 1.0.5 | |
| fof-nightmode | 1.2.2 | |
| fof-masquerade | 2.1.0 | |
| fof-links | 1.0.3 | |
| fof-linguist | 1.0.4 | |
| fof-gamification | 1.4.0 | |
| fof-frontpage | 1.0.1 | |
| fof-formatting | 1.0.2 | |
| fof-drafts | 1.1.2 | |
| fof-default-user-preferences | 1.0.0 | |
| fof-best-answer | 1.1.10 | |
| flarum-statistics | v1.3.0 | |
| flarum-pusher | v1.3.0 | |
| flarum-mentions | v1.3.0 | |
| flarum-markdown | v1.3.0 | |
| flarum-lock | v1.3.0 | |
| flarum-likes | v1.3.0 | |
| flarum-lang-russian | 1.2.0-p1 | |
| flarum-lang-english | v1.2.0 | |
| flarum-lang-chinese-simplified | v1.1.1.1220 | |
| flarum-emoji | v1.3.0 | |
| datlechin-usercard-uid | v0.1.1 | |
| datlechin-scroll-buttons | v1.0.0 | |
| datlechin-copy-links | v0.1.1 | |
| darkle-fancybox | 1.1.0 | |
| clarkwinkelmann-roll-die | 1.0.0 | |
| clarkwinkelmann-post-bookmarks | 1.0.0 | |
| clarkwinkelmann-popular-discussion-badge | 1.1.0 | |
| blomstra-user-filter | 0.1.4 | |
| blomstra-search | 0.1.12 | |
| blomstra-cache-assets | 0.5 | |
| askvortsov-pwa | v3.1.3 | |
| askvortsov-moderator-warnings | v0.6.1 | |
| askvortsov-discussion-templates | v0.8.3 | |
| askvortsov-categories | v3.0.4 | |
| akr-chevereto | 1.0.1 | |
| afrux-top-posters-widget | v0.1.4 | |
| afrux-news-widget | v0.1.1 | |
| afrux-forum-stats-widget | v0.1.1 | |
+------------------------------------------+-------------+--------+
Base URL: https://www.xn--fmrv2rkpbv8uymm.cn
Installation path: /www/wwwroot/www.xn--fmrv2rkpbv8uymm.cn
Queue driver: redis
Mail driver: smtp

And I'm using elastic 7.17.3

luceos

Teddan that elastic version seems to work perfectly fine with CJK, I'm not entirely sure why this is misbehaving then. It would require some extensive research to resemble your set up and attempt reproduction. Are you able to read entries inside elastic using some kind of UX to verify entries are valid?

Teddan

luceos I have access to everything but I dont know how 😢

luceos

Teddan I don't have time right now, but next week I might be able to dive into it for you. It would be easiest for me to access your systems though. Can you DM me on Discord luceos#0001 so that we can get something arranged for next week?

Teddan

luceos sent already 😚

Aaro-n

Teddan luceos You can try to search for Chinese words, visit this link, copy the "测试中文搜索功能" to your forum, and see if you can only search for a single Chinese word, for example, "孔子" only searches for "孔", "曹操" searches for "" 曹" or "操", "blomstra/search" used to be able to search for phrases, but after the latest update, only a single word can be searched.

Teddan

Aaro-n ohhhhh yes . I can search for single Chinese character….. is there any way to support phrase?

Teddan

Aaro-n is the latest version referring to flarum 1.3.0 or the latest version of the the extension?

luceos

Teddan I think he means of the extension. I haven't seen or heard any reports about this though.

Teddan

luceos is this on the future plan to fix? 😢

luceos

Teddan yeah absolutely. I will need to find a solution to that, but I think it's similar to the fix of core.

Teddan

luceos Thank you!! Looking forward to that !

Hari

where can I test this extension without installing it, Is there any website already using it?

i have checked previous posts and went to https://sorhadi.net/ to check
i picked a line from this discussion
Yediğimizi mide hazmederken karın ağrısı when i search using this search not showing any result.

Does flarum or blomstra/search supports full-sentence search?

https://imgur.io/a/LwziYgK

luceos

Hari try this site; forum.mutluanneleriz.com please don't link, https://blomstra.community also has it running but that dataset is much less. Mutluanneleriz contains gigabytes on data in elastic.

Hari

matteocontrini i have almost 30k discussions and it will grow to 70k discussions in the next 10 months. we are using our flarum as a backend application to address user needs, most users will not come to my flarum and perform actions only the admins or mods use it to find similar threads and try to post existing answers so i do not think there will be much user search queries but i am sure this will change after one year. (i will give an interview on this to gXXL after few months, it is a unique case study how we are using flarum)

like mentioned here Hari i am looking for full-sentence matches in post content search

i have 4GB RAM with 2vCPU Plesk setup running for only flarum and 2-3 WordPress subdomains.

do you recommend meilisearch for my need? since you have explored all the possible ways you are the right person to get suggestions from. 🙂

Hari

luceos i have taken this line kaydeden koleksiyonun adını yazsın kaydedeyim and searched, the search is not showing the below mentioned disucssion

forum.mutluanneleriz.com/?q=kaydeden%20koleksiyonun%20adını%20yazsın%20kaydedeyim
forum.mutluanneleriz.com/d/798517-trendyol-koleksiyon-sadece-burada-paylasabilirsiniz/11

luceos

Hari looks like it.

There's definitely room for more tuning and matching mechanisms. I also think it could do with some unit tests. Previous releases of this extension have been mostly about getting data into elastic, improving performance for seeding massive datasets and showing some sensible results.

avandorp

Nice plugin. Is it possible to index uploaded PDF files (e.g. uploaded via FoF/Upload extension) this way? Or is there another way to do that?

luceos

avandorp it is possible, but getting there is not done with the flick of a hand. Flarum itself has no mechanism yet to search files, so that would be step 1. Then one would need to sync uploads (or rather their information) to elastic.

avandorp

Thanks for the fast answer. Naively I imagined that in uploader I would hook up to the File/WasSaved event and via an ElasticSearch ingester I would put the pdf content into the search index. But being new to Flarum I don't see where I would get the info of where the search would have to point to. Where in Flarum would be the right place to do so?

luceos

avandorp Flarum has Search Sources in js. This source would then be added to search, in addition this search source then would need an endpoint that queries elastic for related resources.

Ref:

DiscussionSearchSource: https://github.com/flarum/framework/blob/main/framework/core/js/src/forum/components/DiscussionsSearchSource.tsx
DiscussionSearch endpoint: https://github.com/flarum/framework/blob/main/framework/core/src/Api/Controller/ListDiscussionsController.php

Also look at how I extended DiscussionSearch endpoint: https://github.com/blomstra/flarum-ext-search/blob/main/src/Api/Controllers/SearchController.php
Here we are replacing the old search source, you can mimic this by just adding your own files source: https://github.com/blomstra/flarum-ext-search/blob/aabba9c2d0ff90a15be9ddcd1cc257a497d45607/js/src/forum/index.ts#L16

« Previous Page Next Page »