An extension that handles fulltext search

rafaucau

luceos This extension requires PHP8. Will it be compatible with PHP7.4? flarum/akismet does not work on PHP8 so forums that use Akismet cannot use this extension.

I see that ElasticSearch does not support my language (polish) by default and requires a plugin. It would be nice to be able to set Analyzer language by typing instead of being limited to choosing from a limited list.

luceos

rafaucau This extension requires PHP8. Will it be compatible with PHP7.4?

Nope sorry. In 25 days active support for 7.4 will be dropped, after that it will only receive security updates. And yes we can add a way to add support for more analyzers, I think I'll just make the list extensible from the backend then. I dislike the free choice option as this would cause too much support on the extension if people fill in something that Elastic doesn't understand.

TODO:

discussion titles are not taken into account
offer a button in the admin area to kick off the indexing of your forum (queue is recommended for larger communities)

Hari

i have few questions

is it free? if paid how much it will cost?
do you see rafaucau Telugu lanauge? if not how can i add (edit: as i can see under T there are only two languages and the scroll bar ended there 😢 )

buy just running this command i can install everything?

composer require blomstra/search:*

or do i need to set up an elastic search at the server level?

question-related to related disucssion extension Justoverclock as your extension runs using default flarum search if i enable blomstra/search will it retrieve data from blomstra/search or it will still use default flarum search?

rob006

luceos Nope sorry. In 25 days active support for 7.4 will be dropped, after that it will only receive security updates.

You know that 60-70% of Flarum installations is on PHP 7?

Darkle

rob006 I think he's referring to PHP active support itself, not Flarum support.

rob006

Darkle I know, but cutting off 2/3 of market share is still quite surprising.

tankerkiller125

rob006 Given the extension is primarily designed to run in the Blomstra environment where they have full control over PHP versions and what not I don't find it surprising at all.

Truth be told they could just keep it closed source and available to their customers only, instead they've open sourced it so anyone can use it (assuming the right PHP versions and what not) so I'd say that's already pretty generous.

rafaucau

tankerkiller125 This informations may also be useful to Blomstra. They will probably find a customer who wants to install an extension that does not support PHP8 and use Elasticsearch at the same time.
As I mentioned earlier even one of the official Flarum extensions doesn't work on PHP8 which probably blocks many forums from moving to the newer PHP version.

luceos

Tagged 0.0.8 which fixes the discussion title search.

Todo:

I've added an endpoint that can kick off indexing but: a) a button has to be created in the admin that hits that endpoint and b) that controller needs to execute faster as it iterates over discussions and posts in batches of 50.
Improve relevancy handling (discussion titles before post content for instance).
Allow configuring language analyzers as elastic allows plugins for that or push that post-stable as part of extender logic.

I'll be testing my implementation on my playground community and then push it onto Greenhouse. After that it will be tested on a multi million posts community.

And 0.0.9 which fixes an issue with sending indexing jobs to a specific queue.

010101

luceos Congratulations! I'm sure this hasn't been easy. Or maybe it has.

luceos

Tagged 0.0.14:

added a --continue flag to allow continuing mass indexing.
pushed recreation of index and mappings to --recreate flag.
introduced --only flag to allow indexing specific data types (posts, discussions).
added throttling with --throttle.
added sorting to queries to prevent retries to read different results.
allow forcing the queue to dispatch indexing jobs on.

pkernstock

I think it would be useful to support https://discuss.flarum.org/d/23702-friendsofflarum-discussion-language to Index each discussion based on the set language, to improve search?

luceos

pkernstock I was afraid this question would come and yes that's possible. Understand that this extension stores the post or title string under the content field. But indexing for different languages (analyzers) requires storing that same field again so that elastic understands it. I'd be okay with a PR to get this functionality supported, we don't need it imminently yet though.

luceos

This extension is now at version 0.1.4 and is running on https://forum.mutluanneleriz.com by volkan28 👏 .

Some of the changes:

support for byobu private discussions page
excluding hidden discussions from being indexed
instead of only processing results through elastic when browing the SPA, a direct visit to the page is now also handled by elastic
discussion titles now have a higher score than post content
added comment count for sorting by most replies
and much more..

What isn't working right now is the tags page. But the question is where to move now. The following options exist:

start PR'ing to core to improve search and filtering there so that it will support elastic like it does now
continue extending blomstra/search so that it works with the most used extensions (follow-tags or discussion language for instance).
start working on making the elastic implementation in blomstra/search interchangeable by adding drivers

This extension was meant to resolve the issue of searching fulltext columns. But it is already tackling filtering as well. The Gambit system that Flarum implements is a tough nut to crack, as it would require an implementation in each search/filter driver/extension.

I'd love to hear your opinion on this.

volkan28

Great work. Elastic search is really doing good job and will be better with the feedbacks.

samtto

In Luceos I Trust 😀

MikeJones

Dang, sweet!

CyberGene

I'm trying to make this working with an Amazon AWS OpenSearch from the FreeTier but it does't work 🙁

php flarum blomstra:search:index

In StaticNoPingConnectionPool.php line 64:
                                        
  No alive nodes found in your cluster  
                                        

blomstra:search:index [--max-id [MAX-ID]] [--chunk-size [CHUNK-SIZE]] [--throttle [THROTTLE]] [--only [ONLY]] [--recreate] [--mapping] [--continue]

The domain has public access.

I've copied the domain endpoint, created a master user and configured it in the extension settings and for the index I'm putting just some string, e.g. "forum". What am I doing wrong?

Has anyone managed to run this with an Amazon AWS OpenSearch?

luceos

CyberGene Has anyone managed to run this with an Amazon AWS OpenSearch?

Perhaps the endpoint is wrong, you can use any of the following examples from the elastic.co documentation (don't use an array though):

https://www.elastic.co/guide/en/elasticsearch/client/php-api/current/host-config.html#inline-host-config

matteocontrini

I don't think you can connect to AWS OpenSearch instances with an Elasticsearch client, they have been made incompatible by Elastic on purpose: https://aws.amazon.com/blogs/opensource/keeping-clients-of-opensearch-and-elasticsearch-compatible-with-open-source/

CyberGene

Yeah, I gave up. I tried with the free trial of the ElasticSearch Cloud and it works but I’ve seen something weird: my forum is at https://my-domain.com/forum and some results were returned as https://my-domain.com/forum/forum

luceos

CyberGene can you get in touch on discord luceos#0001, this might be a bug and if that's the case I'd love to solve it.

« Previous Page Next Page »