migrating from Imgur to an S3 compatible provider

navindra

Over a year ago, I chose Imgur as an easy plug and play image host for PianoTell. It has worked well and served its purpose. Unfortunately, I'm now getting reports that Imgur is blocked in the UK.

I need to migrate to a CDN/S3 provider and there are a few things I need to solve:

I need to get and download all the Imgur images linked on PianoTell.
I need to upload all the Imgur images to the new CDN/S3.
I need to replace all the Imgur links on PianoTell with the new CDN links.

For 1, unfortunately the Imgur API isn't being very friendly or is decommissioned, so I'm planning to run a SQL script to get all the links OR export the SQL DB to text and grep for the links. Then use curl to download them. Is there a better way I haven't thought of? (this is done now)

For 2, I just have to figure out the APIs and script the uploads. It will be a straightforward mapping like:

https://i.imgur.com/RFulutX.jpeg -> https://cdn.pianotell.com/imgur/RFulutX.jpeg

For 3, I could use some advice from folks here! How can I update all the Imgur links on PianoTell?

I could somehow do this on the SQL side, though it feels risky. I'd have to run a script that updates everything in place. Are there any example SQL scripts for flarum that do such an update in place? Or I could do it in the text export and reimport the DB -- I've done this before and I know it works.
Maybe I could update the links on the fly with an extension? Although this seems like it might be straightforward, it would be inefficient in the long run and might accidentally rewrite new Imgur links created by a poster.

Any other good ideas folks might have?

Thank you!

GreXXL

navindra it can be done but will involve some custom implementation / scripts for your usecase. We had this question before (with local migration): https://discuss.flarum.org/d/31046-migrate-from-imgur-to-fofupload-local

I don't expect someone will be able to provide this to you as an support answer. If you need assistance and are willing to pay for it you can always open up a discussion inServices.

Umutcan

If you want use a Cloudflare R2, you can read this docs: https://flectar.com/en-US/blog/how-to-use-cloudflare-r2-with-flarum-fof-upload-extension-a-step-by-step-guide FYI @Darkle

KBExit

Umutcan really good for the buck!

navindra

GreXXL Thank you! I'm working my way slowly through this, I'll definitely consider Services if I get stuck. Also happy to share how I get this done if it will be useful for anyone.

Umutcan You read my mind! I've already downloaded 3600 images from Imgur totalling 1 GB and uploaded them to R2 using rclone. Very awesome stuff.

https://cdn-01.pianotell.com/imgur/RFulutX.jpeg is up and working.

For the short term, I will try to rewrite imgur URLs on the fly and then think about updating the DB later.

navindra

Here are the cliff notes for anyone who needs to migrate from Imgur to S3.

Disable Imgur uploads on your forum and configure S3 instead:
https://flectar.com/en-US/blog/how-to-use-cloudflare-r2-with-flarum-fof-upload-extension-a-step-by-step-guide

Download all your assets from Imgur. There are many ways of doing this.
a. Dump your DB to SQL:

mariadb-dump --host=pianotell-database --single-transaction --skip-dump-date --lock-tables flarum_pianotell > pianotell-dump.sql

b. Now get all the Imgur links:

grep -o "https://i.imgur.com/[[:alnum:].]*" pianotell-dump.sql > imgurlinks-0.txt
sort imgurlinks-0.txt | uniq > uniq-0.txt

c. Now download all the images. I used a pretty high sleep because I didn't want Imgur to throttle me. Feel free to adjust:

mkdir downloads-0
cd downloads-0
xargs -n 1 -I % sh -c "{ curl -O  %; sleep 5; }" < ../uniq-0.txt

Now set up rclone:
https://developers.cloudflare.com/r2/examples/rclone/

Now upload your images to S3:

cd downloads-0
rclone copy . r2:pianotell-cdn-01/imgur/

Now update Flarum extend.php to rewrite the Imgur links to the S3 URL:

use Flarum\Extend;
use Psr\Http\Message\ServerRequestInterface as Request;
use s9e\TextFormatter\Configurator;
use s9e\TextFormatter\Parser;
use s9e\TextFormatter\Renderer;

return [
    (new Extend\Formatter)
        ->render(function (Renderer $renderer, $context, $xml, Request $request = null) {
            return str_replace('https://i.imgur.com/', 'https://cdn-01.pianotell.com/imgur/', $xml);
        })
]

Please let me know if there's a better way of doing Step 5.

I do intend to update the links directly in the SQL DB eventually, but this is a risky operation.

navindra

I hated the idea of the database having bad data. This is how I completed the final task:

/* take a backup and necessary precautions */
mariadb-dump --host=pianotell-database --single-transaction --skip-dump-date --lock-tables flarum_pianotell > before.sql

mariadb

CONNECT flarum_pianotell;

/* just looking around. these were the only tables I cared to update. */
SELECT * from posts WHERE content LIKE "%https://i.imgur.com/%" limit 10;
SELECT * from drafts WHERE content LIKE "%https://i.imgur.com/%" limit 10;
SELECT COUNT(*) from posts WHERE content LIKE "%https://i.imgur.com/%";
SELECT COUNT(*) from drafts WHERE content LIKE "%https://i.imgur.com/%";

/* now do the dangerous stuff. */
UPDATE drafts
SET content = REPLACE(content, 'https://i.imgur.com/', 'https://cdn-01.pianotell.com/imgur/')
WHERE content LIKE "%https://i.imgur.com/%";

UPDATE posts
SET content = REPLACE(content, 'https://i.imgur.com/', 'https://cdn-01.pianotell.com/imgur/')
WHERE content LIKE "%https://i.imgur.com/%";

exit;

/* another checkpoint for posterity */
mariadb-dump --host=pianotell-database --single-transaction --skip-dump-date --lock-tables flarum_pianotell > after.sql;

/* look at the diff for a sanity check */
code --diff before.sql after.sql

Probably pretty risky stuff, but PianoTell survived.

Umutcan

navindra Thanks for sharing, you'r amazing 💫