FriendsOfFlarum upload, the intelligent file attachment extension

Wlork

Hello,

How I can increase the maximum image size ? In Admin, I increase the value but it still at 2 MB in practice....
For information, I use IMGUR as "storage" and the site authorizes until 20 MB....

OfficialAudite

Wlork You have to make your php setthins to take higher and then adjust trough admin.

Wlork

OfficialAudite Thank you, it work 🙂

CyberGene

@luceos can you explain in more details how the new functionality works and probably update the first post in this discussion? Currently the commands are in a post somewhere in this discussion and it's hard for me to find it among 2000+ posts (BTW, here's where load while scrolling for Flarum has its downsides, it's just too awkward and for some reasons while loading the browser skips some posts, so it's not even smooth).

In particular can you explain what the --map option does? My understanding is you should execute it only once after you upgrade to the new dev in order to map the existing files and posts (and the mapping will be stored in the database for future use), thus there will be no need for second execution of the --map, is that correct? If not, does that mean that I should always combine --map with --cleanup? If I execute only --cleanup with no --map, what will be the result? Are the files physically deleted from the corresponding storage, e.g. AWS? I remember there were some errors with aws-s3 vs aws_s3 in the DB or something like that and you said you're going to implement a proper migration. Is that ready now? Can I safely execute the cleanup logic on my prod forum or is that functionality still in beta?

Again, a better description of the algorithm and the commands would be very helpful since I am confused and still worried to use it. I haven't done so anyway, because of the DB discrepancies and the errors, as well as fear that the functionality is still in beta.

P.S. To add to the above, executing --cleanup without --map results in the following warning:
Are you sure you want to clean up without mapping first? (yes/no) [no]:

So, what's the difference, what will happen in the two scenarios, with and without mapping? Am I gonna lose posts/files?

P.P.S. So, I still get the In Manager.php line 85: No adapter configured for aws_s3. Is it safe to update my DB records? But I think this should have been a migration since I have been using only official releases of extensions. And besides, all my new uploads keep on using the aws_s3. What extension/code is inserting records in fof_upload_files table with upload_method set as aws_s3 and why is the extension expecting it to be aws-s3 instead? Not sure that's OK.

luceos

CyberGene I've added some clarification to the OP and I've pushed two migrations to help your adapter settings being moved over. This applies to both the mime type configuration setting and the uploaded files. You can give this a spin with the 1.3.3-beta's:

composer require fof/upload "*@beta"

That should install 1.3.3-beta.3.

CyberGene

luceos thank you very much, I will give this a try tonight 👍🏻

luceos

CyberGene did it resolve it for you?

CyberGene

luceos yes, I installed the beta extension and ran it once against my forum with mapping and cleaning (but no force) and went through all the files just to review them and all seems good, no error this time, thank you! I am still waiting to actually start cleaning up files because I informed the forum users about what's going to happen, so in case someone has some uploads that they didn't use in any post, they should back the files up. I thought I would wait to perform the actual clean up before sharing some feedback but even now all seems good and there's no error anymore, so I can confirm, thank you again 👏🏻

Just to confirm, a few questions:

Is the mapping functionality capable of detecting orphaned files from before the new development was introduced? By orphan files I mean those files that are not used in any post and whose media library entities have been deleted by the user? In my understanding what remains is just a file on the storage that nobody knows about?
If the answer to the above questions is no, then is there a way to detect such files? I guess it's a bit tricky since such a functionality would have to list all the files on the storage and check whether there are links to them in any post.
With the latest version of the extension, if someone uploads a file and never uses it in a post and then deletes the media entity, what happens? I would assume that in such a case the file is still mapped correctly, so that it can be cleaned up without the need to explicitly perform a new mapping, right?

luceos

CyberGene Is the mapping functionality capable of detecting orphaned files from before the new development was introduced? By orphan files I mean those files that are not used in any post and whose media library entities have been deleted by the user? In my understanding what remains is just a file on the storage that nobody knows about?

If no entry is available in the database, then yes, Upload will not know about it and cannot delete those files sadly.

CyberGene If the answer to the above questions is no, then is there a way to detect such files? I guess it's a bit tricky since such a functionality would have to list all the files on the storage and check whether there are links to them in any post.

Possibly, if a dedicated S3 bucket was used we could create an index of the files that are known and go over all files in the S3 bucket to delete that are unknown. It will be a painful process as this will certainly hit api limitations on large buckets/files.

CyberGene With the latest version of the extension, if someone uploads a file and never uses it in a post and then deletes the media entity, what happens? I would assume that in such a case the file is still mapped correctly, so that it can be cleaned up without the need to explicitly perform a new mapping, right?

Yes it will be deleted if you use the command.

Please understand that this is a first step into the direction I wished I had taken at the inception of Upload, namely to keep track and remove orphaned files. If at any point these files remain, we hope to find a solution to tackle those use cases too.

CyberGene I encountered again the error No adapter configured for aws_s3. So, the migration fixed the existing entities, however the newer ones keep on being stored with the aws_s3 adapter. Can I perform the migration again to fix the newer entities or a migration is a one-off thing that won't happen again unless I upgrade an extension?

If you remove the db entry with the same name from the migrations table, you can. I would love to hear whether it works the second time, because it should not happen.

delete from migrations where migration = '2023_08_20_000000_alter_uploads_s3_adapter.php';

CyberGene

An update (sorry to post very shortly after my previous post but I don't want this to be missed).

@luceos , I just executed again a php81 flarum fof:upload --map --cleanup and when I reached some recent files from the past days I encountered again the error No adapter configured for aws_s3. So, the migration fixed the existing entities, however the newer ones keep on being stored with the aws_s3 adapter. Can I perform the migration again to fix the newer entities or a migration is a one-off thing that won't happen again unless I upgrade an extension?

CyberGene

luceos If you remove the db entry with the same name from the migrations table, you can. I would love to hear whether it works the second time, because it should not happen.

delete from migrations where migration = '2023_08_20_000000_alter_uploads_s3_adapter.php';

Well, I did that and executed a migrate command (and the above migration was performed), however the error is still there. So, I removed the DB record above but also the other one 2023_08_20_000000_alter_uploaded_files_s3_adapter

And after a new migration the files are now fixed. Maybe you swapped which migration DB record to remove but just in case one can delete both.

Now everything seems to be working correctly, thank you very much 👏🏻 I'll also perform a forced cleanup of everything soon which will be the moment of truth 🙂

luceos

1.3.3

Okay, finally tagged. It has seen many beta's, but it also blocked some features @clarkwinkelmann was working on.

This releases adds the first logic to map uploaded files to posts and a basic command to delete files not or no longer used in posts. You can read about this logic in the OP.

This command is not obligatory, nor is mapping uploads to posts. But I highly recommend you to use the logic at least for mapping using php flarum fof:upload --map. Once you've run that once, Flarum will keep track of the new uploads automatically.

johnvanhulsen

luceos This command is not obligatory, nor is mapping uploads to posts. But I highly recommend you to use the logic at least for mapping using php flarum fof:upload --map. Once you've run that once, Flarum will keep track of the new uploads automatically.

Great. I've tried it but I think there is a small bug inside.
We use DB_TABLE_PREFIX="flarum" in our .env file (and flarum as prefix in the DB 🙃).

It seems that the prefix is placed twice in the query. I've got the following error:

php flarum fof:upload --map

In Connection.php line 712:
SQLSTATE[42S02]: Base table or view not found: 1051 Unknown table 'flarumflarumfof_upload_files' (SQL: select `flarumflarumfof_upload_files`.*, group_concat(distinct posts.id) as matched_post_ids from `flarumfof_upload_files` left join `flarumposts` on `flarumflarumfof_upload_files`.`actor_id` = `flarumposts`.`user_id` and `flarumposts`.`content` like CONCAT('%', flarumfof_upload_files.url, '%') group by `flarumflarumfof_upload_files`.`id` order by `flarumflarumfof_upload_files`.`id` asc limit 1000 offset 0)                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                       

In Exception.php line 18: 
SQLSTATE[42S02]: Base table or view not found: 1051 Unknown table 'flarumflarumfof_upload_files'  
                                                                                                    

In PDOConnection.php line 82:
SQLSTATE[42S02]: Base table or view not found: 1051 Unknown table 'flarumflarumfof_upload_files'  

fof:upload [--map] [--cleanup] [--cleanup-before [CLEANUP-BEFORE]] [--force]

luceos

CyberGene Now everything seems to be working correctly, thank you very much 👏🏻 I'll also perform a forced cleanup of everything soon which will be the moment of truth 🙂

🙏

CyberGene

luceos So, a heads up, I just uploaded some new file and it still appears in the fof_upload_files table with upload_method aws_s3 and not aws-s3 🙂 I'm on the just released fof-upload 1.3.3.

Honestly not a problem, I kind of see how things work and can update those records. So, as I said, just a heads up if you need to fix this. Not sure if anyone else is having that problem, seems nobody else complained.

Darkle

Upgraded and did another cleanup with an R2 bucket, all good! Although I can confirm what CyberGene mentions, the migration updated the previous files with aws_s3 to aws-s3, but the newly uploaded files continue to add aws_s3.

By the way, I don't remember it very well (I think it was discussed at some point), but if I remember correctly this system maps the files internally, so now that it's stable it will eliminate the need to show the id in the filename?

luceos

Darkle By the way, I don't remember it very well (I think it was discussed at some point), but if I remember correctly this system maps the files internally, so now that it's stable it will eliminate the need to show the id in the filename?

Yes it stores the url in the table, so it tries to find that in posts.

Darkle Upgraded and did another cleanup with an R2 bucket, all good! Although I can confirm what CyberGene mentions, the migration updated the previous files with aws_s3 to aws-s3, but the newly uploaded files continue to add aws_s3.

Aha I see, I will look into it.

maxmoon

Check all uploaded files

Is there a way to see all uploaded files (from all users) in one go?

As an admin I would like to be able to see what was uploaded on my server and delete it fast.

How can a user delete files?

If I try to delete a file as a user in my library, a dialog says it will only be hidden. Why? Even if I try to delete something, what wasn't used in a post, it will be only hidden.

This is fatal! Because I really don't want to have ghost/garbage data on my server.

Is compression possible

The crop avatar extension has a builtin compression. It would be nice to have something like this for the images, which gets uploaded, too.

Delete unused files frequently

Is it possible to delete files, which aren't used in the forum with a cronjob?

Illegal usage of file uploads

The possibility to use the upload can be divert from its intended use. People could upload illegal/copyrighted/pornographic/pedophile media without the knowledge of the admin and use it somewhere else to share it.

Another bad thing is, that a new user could upload stuff, even if first approval is active. For uploads there is no approval needed and can be done while composing a post.

Is there a way to control or check this? It would be already helpful if the things work, which were mentioned above (e.g. delete unused files automatically).

Thank you 🙂

Darkle

maxmoon I am sure someone will be able to answer this in more detail than I can, but to try and solve it, here is ....

Check all uploaded files

If you mean an interface that allows you to see all the images uploaded by all users, that doesn't exist, although if you use external services to host images, such as S3 bucket... etc, it's quite likely that you could see the uploaded images at a glance from there.

How can a user delete files?

I don't know the reason for this, I think there was an issue on github about this. Anyway, now with the cleanup command, if the user has actually removed the image from the post and it's not being used (I'm not talking about the media manager), you can remove the unused garbage, I've already removed a lot of GBs since the beta came out.

Is compression possible

You can crop the image to a maximum limit, I don't know if the library used also applies compression, but there doesn't seem to be a specific option.

Delete unused files frequently

This is possible since the last version with the command:

php flarum fof:upload --map --cleanup

With this you can delete all unused files, you can even specify date ranges if needed.

Nothing would prevent you from adding a cronjob with this command, although it would be important to add --force as this way it would not require image-by-image confirmations.

php flarum fof:upload --map --cleanup --force

Illegal usage of file uploads

Honestly, I have no idea if there is a tool for this, although as an opinion, considering that on big platforms (social networks...) there are often unpleasant accidents with this kind of thing, I guess that even these top-notch filters don't work very well. I run a Flarum for all ages, I use spam prevention so that if any post contains a URL it is put under approval, I think in this sense there is nothing as effective as manual moderation.

luceos

johnvanhulsen yeah this was reported on GitHub and I found the reason for it. Will patch this asap.

« Previous Page Next Page »