Setting up a queue to process photos (fof/upload)

BurnNoticeSpy

G'day again. I'm trying to process up to 40 photos on a single API upload request to the fof/upload extension, and that is actually working quite well - but i also run some image processing on each photo and overall this slows down the response to the client. I want to speed up this response.

I know i need to set up a queue to offset this initial workload and perhaps only process 6 images initially and delegate the rest until later (within 30 minutes the job should be completed though).

Should i use the bloomstra/database-queue or bloomstra/flarum-redis extension? Which one, why?
How can i keep track of the count of the uploaded files to decide whether to process or add to queue?
What should i do to ensure the uploaded file stays connected to the discussion if processed later? (I currently only store the image paths in a custom images column).

Sorry if i'm asking for a lot. Anything helps!

luceos

BurnNoticeSpy the database one is the easiest to set up. It can run as a cronjob and not necessarily as a daemon.

BurnNoticeSpy

luceos Ok, thanks mate.

I installed the extension, ran crontab -e and put * * * * * cd /var/www/websiteroot/flarum && php flarum queue:work --stop-when-empty.

I also tested another command to simply print to the terminal to confirm it was running every minute and it indeed was working well.

Now I've set up the backend to push a task into in the queue and it's being processed which is awesome but it's processing it instantly instead of waiting until the next interval (minute).

I might have ran php flarum queue:work a few times before i setup the cron job.

How do i monitor my queue logs when cron job triggers? Maybe i need to stop something already running???
It shouldn't be processed near instantly right?

luceos

BurnNoticeSpy Maybe i need to stop something already running???

It definitely feels like something is continuously running. On linux you can usually check that with ps -aux | grep queue or something similar.

BurnNoticeSpy How do i monitor my queue logs when cron job triggers?

The queue should either log into the default flarum logs under storage/logs or another file in that same directory.

BurnNoticeSpy

Well actually, i think the queue driver is working well, but for some unknown reason cron won't run flarum's php flarum queue:work --stop-when-empty command automatically.

This in spite of the fact that...
a) the identical command runs when i manually put it in the terminal
b) cron is 100% enabled and working with another command wall -n Display this message to all terminals. every minute.
c) no errors are being logged in flarum or anywhere else that i can find
d) i've tried running it as sudo
e) the queue is loaded up with a task before the interval runs (confirmed when ran manually)

Wtf?????????? My intro class to cron and queues is not going well :/

clarkwinkelmann

BurnNoticeSpy you could try piping the output of the CRON command to a log file (php flarum queue:work --your-params > /path/to/log/file.txt 2>&1), to allow you to read what the php flarum executable says before quitting.

If there's nothing in the official Flarum log file, maybe it's a permission error (you should be able to confirm with the method in the previous paragraph)? For optimal results you should make sure the flarum executable runs as the same user as when you use SSH. This might be a setting in the CRON, or you could use the sudo -u syntax to change the user the command runs as.

luceos

BurnNoticeSpy can you post here which command you added to the crontab and which user that crontab belongs to?

BurnNoticeSpy

clarkwinkelmann

You're right! It was a permission error related to the flarum log. I actually got some output from cron along the lines of '...The stream or file flarum-2022-10-19.log could not be opened in append mode: failed to open stream: Permission denied in...' from my queued task's php file. My task was simply trying to log a message to show it had been successfully ran while i monitored tail -f flarum-2022-10-19.log and waited for cron to trigger.

So it might have been working but it failed on the first and only line of log() since the task was not very substantive (yet), but from my perspective no 'success' message was logged, and it had worked when ran manually and i didn't know it had any errors while running. I had also outputted to the flarum log from my php controllers fine, so it seemed unlikely to be permission related.

But no need to look further, i learnt a bunch already and it's working now after adjusting storage folder permissions.

Thanks @clarkwinkelmann and @luceos

luceos

BurnNoticeSpy You're right! It was a permission error related to the flarum log.

The reason this happened and the reason it might happen again is because you are probably running the queue worker as root. The cronjob should be configured for the user the website runs as, usually www-data. The command for that would be: crontab -u www-data and you'd add the command in there.

BurnNoticeSpy

luceos I'll keep that in mind, thanks @luceos.

Btw I've fought for a few hours now to overcome errors like this

[2022-10-19 15:49:01] flarum.ERROR: InvalidArgumentException: Database connection [my_database] not configured. in /var/www/mywebsite/flarum/vendor/illuminate/database/DatabaseManager.php:161

I'm just trying to move my (new ImageManager())->make(... working code from my ModifyUploadedImages.php file which currently is listening to fof/upload WillBeUploaded $event over to my queue worker file which extends AbstractJob. But i keep coming up against the error above and i'm sure it's something simple...but it's driving me insane. Any ideas??? I was trying to make good progress ugh :/

luceos

BurnNoticeSpy try running the queue worker manually with -vvv to get a full trace of the error.

BurnNoticeSpy

luceos Thanks @luceos. I have come to learn that error was just a symptom and not the root cause of the problem, i eventually got some logs that hinted at the root cause.

Most of my problems have been permission related ranging from unable to log(), unable to read image sources errors, empty image files etc. So turns out your advice of using crontab -u www-data is really important to follow and also file permission being set right!

But after many days of hard work my fof/upload queue is set up and working great. I had no other options but to make a few minor changes to the extension UploadHandler.php to give my event listener more control to override the upload process. I also had to introduce a discussion id parameter to be included on each upload to link my uploads to discussion's photos column since it would have no idea what it belongs to. Maybe i'll fork fof/upload to manage the changes since yes i know they will be overritten.

Bonus problem was solved after HOURS of troubleshooting and confusion after finally realising that fof upload deletes the temp file if an error is returned. I was trying to access it after throwing an exception (aka my "i'll take over from here" command). Good times...