Diff for Flarum

rob006

Kylo We need to store all of the old post's content in the database rather than storing changed parts (diffs) only.

IMO it is still better and more pragmatic to store full content of old revisions, instead of calculated diffs only. It should be less problematic in future when some more invasive changes should be made in diff format and makes more sense in context of removing revisions. Right now if you if you have use case like this:

I wrote my post. (revision nr 1)
I edited this post and put some sensitive data in it. (revision nr 2, diff nr 1)
I realized what just happened and removed sensitive data from my post. (revision nr 3, diff nr 2)

To remove this sensitive information form editions history, I need to remove all two diffs. If you store raw revisions and calculate diffs on request, I could just remove revision nr 2. And you could still generate usable diff between revision nr 3 and revision nr 1.

It will be also much easier to create "revert" feature (you just need to copy content from revision to post).

Kylo

rob006 You've got a point but i still think it will cost too much for a rare situation. Imagine a board with 10k posts and all of them have 4 revisions. This means 40k additional posts just to track the changes that been made. I would rather force users to remove their diffs which have sensitive informations.

On the other hand, each approach has its own flaws. If the post contains small amount of words, then it'll be much better to store old version's full content for the sake of storage volume. But if it contains larger text, it will be enough to indicate changed parts only.

rob006

Kylo I don't think that storage should be a concern at all. Most of posts is never edited, large revisions table should not affect forum performance and size of this table should be a fraction of size of the posts table. This is also opt-in feature - if someone is concerned about database size, he could just do not use this plugin.
And for short posts (1-3 paragraphs - probably the most common case) current format is less efficient, since you need to store paragraph before and after the change + neighbor lines (which may include the whole post) + JSON overhead. If you really want to optimize size of this table, I would rather suggest using COMPRESSED row format for it - it will transparently compress post content before saving on disk, which may result 50%+ data size reduction.

jordanjay29

Kylo I prefer this approach. The smaller storage size and efficiency is a better long-term benefit than a few instances of preserving diff history integrity over a short term. If one or two posts needed to have their diff histories compromised due to human error, that's not a big deal, but if a site becomes slow or bloated because of this extension it surely is a big deal.

I'd probably just test that the removal of a not first/not last diff wouldn't affect the flow of diff history, and leave it up to individual sites to manage handling a use case with sensitive information.

Pollux

rob006 Since revisions are stored in separate table, which is accessed only if someone opens popup with list of posts revisions, it shouldn't matter from performance perspective.

I am with @rob006 here. Storing full revisions is far more future proof. The gain in performance* (if there is any at all) will not be worth the potential hassle. Having the revisions in a separate table is the way to go.

Kylo Imagine a board with 10k posts and all of them have 4 revisions.

This is an unrealistic assumption. From my experience, one revision for every post (on average) is already on the high side of a realistic expectation, most posts never get any edits.

jfcherng Some thoughts for storing the whole post:

Only keep the latest N revisions in the DB

Auto (or provide a manual way) delete outdated revisions (like 1 year ago) from DB

Only provide revisions for the main post but not replies (like WordPress)

In my opinion none of these measures is needed, the first two wouldn't be very effective either.

Edit: * by storing only the diffs

Kylo

rob006 i'll consider your suggestions. I'm curious about other's thoughts on this 🤔

rob006

Kylo You can check how others implemented this feature. All examples I know (WordPress, Drupal, Stack Overflow, Redmine, AFAIK even MediaWiki) store complete revisions and computes diff on the fly.
I also have example from phpBB forum - with database of size over 1.5GB, posts revisions table had about 30MB. I know it could look scary when you think about it from "10k posts and all of them have 4 revisions" perspective, but this is unrealistic example. It really looks like worrying about storage size is just premature optimization which will create problems and limitations in future.

rob006

jordanjay29 but if a site becomes slow or bloated because of this extension it surely is a big deal.

Since revisions are stored in separate table, which is accessed only if someone opens popup with list of posts revisions, it shouldn't matter from performance perspective. Even if this table had terabytes of data, fetching few records by post ID should be super fast (as long as table has correct index).

jfcherng

If the size of DB doesn't matters, saving the whole post for every revision is definitely better because of future-proof flexibility. It will also make your PHP plugin codes less complicated since you no longer need an intermediate representation (i.e., the JSON renderer) but just compute the diff with all diff/renderer options changeable on-the-fly (and makes your codes have better compatibility with my jfcherng/php-diff lib, or easier to switch to another lib).

Some thoughts for storing the whole post:

Only keep the latest N revisions in the DB
Auto (or provide a manual way) delete outdated revisions (like 1 year ago) from DB
Only provide revisions for the main post but not replies (like WordPress)

Kylo

Loved the brainstorming but i think we're thinking a little bit 'small' here. Not everyone using this software just to say 'thank you' or 'how was your day?' to each other. Wiki posts was one of my main reasons to build this extension. 10k posts with 4 revisions is not so unrealistic assumption when you take them account. I don't know, maybe i could say 100 wiki posts with 3 revisions for each just to be more realistic.

rob006

Kylo Wiki posts was one of my main reasons to build this extension.

Then wiki post is the main reason why you should store complete revisions and generate diffs on the fly. Reverting changes and comparing non-consecutive revisions (like show differences between revision 2 and 7) is must-have feature for even the simplest wiki. Current approach will make it nearly impossible.

tankerkiller125

Something I do want to note is that Flarum handles large amounts of post and discussion quite well, as an example one of the largest communities running flarum currently has more than 33 Million discussions/post on it and Flarum is still chugging along as if it was a fresh install. Now I'm sure they have quite a bit of infrastructure behind all that such as load balancing and other technologies, but in the end it's still just PHP, MySQL and a Web Server.

ChadJessen

tankerkiller125 What community is that? I would be interested in seeing that.

Kylo

rob006 It's possible with the line numbers actually but the backend will be sooo complicated. Alright, i'm convinced. Next version will store the whole content instead of diffs. I'll add some other options like "just store main post revisions" and "don't mark the post 'edited' for X minutes after its creation time (or its previous revision's creation time)". I'll also implement a lazy loading to the revisions dropdown.

rob006

Kylo It's possible with the line numbers actually but the backend will be sooo complicated.

It is possible with diff format, unless someone removes one diff from post history 😃

matteocontrini

Kylo don't mark the post 'edited' for X minutes after its creation time

I would really love to have this feature provided as a standalone extension as well 😃

Pollux

Kylo I'll add some other options like [...] "don't mark the post 'edited' for X minutes after its creation time (or its previous revision's creation time)".

I agree with matteocontrini, this I will appreciate a lot.

Kylo

Quiet Edits for Flarum

As I promised in Kylo, this is a preparation for next version of my Diff extension. I'm not sure if I picked the right title for this extension 🤔. Anyways, edits made within the grace period immediately after posting will not be considered as formal edits. You can also ignore whitespace and case differences independently from the grace period.

And again, it's based on @jfcherng's diff repository.
It raises new events for developers, called PostWasRevisedQuietly & PostWasRevisedLoudly

Settings

Requirements

You can check your php version by running php -v and check if iconv is installed by running php --ri iconv (which should display iconv support => enabled).

Installation

composer require the-turk/flarum-quiet-edits

Updating

composer update the-turk/flarum-quiet-edits
php flarum cache:clear

Usage

Enable the extension. The grace period is 120 seconds, whitespace and case differences will be ignored by default.

Links

JoshyPHP

With regards to storage, as much as I'd like to store precomputed diffs I would definitely recommend going with the full post in plain text (not the XML form.) The main reason is that if you store a diff and the text it's based on gets modified in any way (e.g. manually or via another extension) then your diff won't apply cleanly or maybe at all.

If you need to deal with outliers, I'd suggest archiving old revisions as compressed blobs. For example, if a post reaches 15 revisions, remove the first 10 and compress them together as a single object. Since the revisions will be substantially similar, they will compress very well. You could have that as a cron job. The downside is that it makes accessing old revisions more complicated.

Kylo

JoshyPHP What about storing all of the post's revisions as merged & compressed BLOB instead of checking the old ones?

Littlegolden

I have edited this post five times but it always shows no edit history. . . There was an error in the console and Flarum did not generate an error log for this.

HD3D

Littlegolden Facing the same issue, It always say No Revisions

Kylo

Littlegolden

HD3D sorry, i am aware of the issue and will be solved in next release. There's a typo in PHP codes

theportablegeek

HD3D Did the update solve this for you? I am still running into the issue....

« Previous Page Next Page »