Database Queue - the simplest queue, even for shared hosting

poVoq

luceos Go into your hosting control panel and set up the following task to run every minute

Some shared hosters limit the interval you can run scripts to 30 minutes or so. What would be the impact of that when running this plugin? Just that the emails get dispatched only every 30 minutes or would something more important be effected?

What about registration emails? Could those still be dispatched immediately & independently from this queue plugin?

tankerkiller125

poVoq if it's anything like other queue plugins, all emails and things of that nature would get delayed by 30 minutes. And if you have a lot of emailing to do the queue could timeout before it finishes meaning that not all the emails are sent (so so people wait an hour, then hour and half, then two, etc.)

luceos

poVoq Some shared hosters limit the interval you can run scripts to 30 minutes or so.

What tankerkiller125 means is, even though the database queue is a queue, using the cron won't keep the worker running continuously. As such queue tasks are indeed only processed at the latest each 30 minutes if your hosting provider doesn't allow running faster.

Personally I don't think using this database queue driver on a hosting provider that limits cronjob interval to 30 minutes is the right solution. But to those that can configure it to run every minute, or at least every 5 minutes, it should be a really great intermediate solution to speed up their community.

tankerkiller125 And if you have a lot of emailing to do the queue could timeout

With this extension that isn't really possible. If the worker receives an enormous pending task list the worker will keep working on that list, after 30 minutes a new worker will be dispatched which will also start working on the task list. At some point all tasks are completed and each worker will terminate. Some hosting providers might forcefully kill duplicate workers (I certainly don't hope so), as the last task wasn't finished the new worker will simply continue with that one.

I must say, this queue driver is not tested and all I've said above is based on practical experience using cronjobs, scheduled tasks and similar environments. There might still be situations things don't go as planned, if that is the case I would love to think along to resolve that.

rob006

luceos With this extension that isn't really possible. If the worker receives an enormous pending task list the worker will keep working on that list, after 30 minutes a new worker will be dispatched which will also start working on the task list.

Shouldn't there be a protection to prevent spawning concurrent workers? Then you could probably skip --stop-when-empty flag and allow worker to run continuously.

luceos

rob006 yeah, some providers might kill duplicates or even prevent them.

luceos Some hosting providers might forcefully kill duplicate workers (I certainly don't hope so), as the last task wasn't finished the new worker will simply continue with that one.

Removing the --stop-when-empty indeed makes sense if the provider prevents duplicates and doesn't time workers out.

rob006

luceos yeah, some providers might kill duplicates or even prevent them.

I'm not even sure how they could automatically detect duplicates (since they're not duplicates - every process is doing its own job), but avoiding concurrent workers will prevent multiple problems, like race condition issues or draining all resources by workers that spawns indefinitely.

luceos

rob006 if people decide to use this, I'd love to hear their experiences. I'll probably not invest too much time on this for several reasons:

I've build this (simple) extension to improve performance on communities that are active, but not large.
I've build this to generate a little goodwill on behalf of Blomstra. But we are not using it and neither am I nor discuss.
The core team agreed to merge this alternate, simple driver into core. It means I'll have to mimic some of the logic the mail driver has. In addition we decided to only show these advanced features based on certain activity thresholds. More about this in the dev diary soon.

shebaoting

Can it exist alongside blomstra/flarum-redis?

luceos

shebaoting it can, but only one queue driver will be active at the same time. Blomstra/flarum-redis allows you to disable the queue, this extension (the database queue) is enabled when enabled in the admin area.

010101

I have this set up and it works well. Thank you 🙏

poVoq

Could you also add a public link to this so that another server could trigger it?

luceos

poVoq if it connects to the same database yes. And maybe you need to use a different cache mechanism but that is optional.

poVoq

luceos Can you explain a bit more? I was thinking about writing a small php script that can be accessed from the webserver publicly that triggers this plugin, as the shared host I am running this Flarum on only allows scheduled tasks at every 30 minutes. But I have another server that I don't want to put this forum on, which could just ping the php script regularly.

luceos

poVoq set up an identical flarum install on that other server that connects to the database of your production site. Run the worker there. There might be some unforeseen issues (cache flushing for instance when changing extensions) but other than that it should work as is. Note that you will need to update two installs if you update your production website.

poVoq

luceos Hmm, that would not work as the database isn't accessible from the other server.

But why would running it through a php script be any different from running it from a cron?

luceos

poVoq I don't know what your PHP script does and honestly if you hit a web endpoint from a Cron on another machine you will possibly just hit timeouts continuously. I think it's too unreliable. The hoster might interpret the activity as unwanted too for instance.

Header_Footer

Sir Thank You so much for this extension, It boosted my posting speed before when i click post reply it takes at least 5 seconds and now in milliseconds post is posted.

Sir as I can see it's in beta stage so is it safe to use it in production and also running a cron every minute will add load on server or not, I am using a 4 core cpu? 16 gb ram

luceos

Header_Footer the cronjob that checks for background tasks to process does not add any noticeable load to the server by itself. When processing tasks it actually does so efficiently while taking that load away from the interaction that causes those tasks to be fired. So netto/bottomline the use of a queue is more efficient than without it.

Hari

luceos unable to enable DB queues extension, do you think it is because I'm using MySQL 5.6?

Edit : for now I want to try againRedis queue using corn job like mentioned here rafaucau

I will keep you posted. thanks for responding. 🌼

luceos

Hari check the logs for the actual error or enable debug. I cannot tell what the problem is with just the fact that an error ocurred.