Optimization Techniques For Flarum

Wombat

Franz Not anymore, haha. But, it's object literal that is sent across the pipe and it would still be fine. Flarum parses that JSON anyway so it becomes literal, so it will work either way. This way, you're just not sending quotations around property names and keys. (We're still using quotations around property values, which is required).

I use this method when loading in player's inventory items from my nodejs server via websockets. Sending object literal data across the pipe from a trusted source (which this case is), is an amazing bandwidth saver. Same thing can be done through the /api/ calls.

Franz

The problem I see is that this API may become a public API, and I wouldn't want to require people to have to go hunt a compliant JSON parser just to use our API.

Wombat

Franz Yeah, maybe... this could be created by me as an Extension of some sort if possible. The Bandwidth Saver I shall name it!

TLDR: First option: Remove unnecessary quotations around property / key names from ajax calls basically.

Second: If the user wanted to go to an extreme case.. use alphabetical naming json to be sent across the pipe (they would have to edit flarum's js core to match the property names though). This could all be done as an extension, but would kind of...screw everything up because if they were to install another extension or if flarum delivered an update, or whatever, shit's going to hit the fan within the code, and the user would have to manually change property values to match, etc just a big hassle. First option I recommend though! 🙂

JoshyPHP

Franz You could make the payload format configurable, while keeping JSON as the default. Same API, just add a "format" parameter to the request.

I've never used MessagePack but I've heard good things about it.

Wombat

JoshyPHP

That seems cool, but the only issue I see is, is there a javascript parser for it? (If the msgpack is sent across the pipe we need the browser to read/convert it somehow)

JoshyPHP

There's at least one implementation but I don't know what it's worth.

That said, I think the low-hanging fruit in terms of network efficiency is enabling gzip. It doesn't look like it's enabled for JSON response.

Toby

I think this is a great example of premature optimization.

Firstly, I couldn't seem to replicate the magnitude of savings you claim to have made by removing double quotes around keys. My method was to use the following find/replace regex in Sublime Text:

   Find:  "([^"]+)":
Replace:  $1:

I tested this method on two payloads, and then gzipped the results:

Discussion List

With quotes: 41067
Gzipped: 9931

Quotes removed: 38755
Gzipped: 9868

Word association game discussion

With quotes: 32015
Gzipped: 4260

Quotes removed: 27903
Gzipped: 4210

Conclusion

Removing double quotes from keys introduces extra complexity by breaking away from standards. It produces negligible savings, especially after gzipping.
Using alphabetical naming is essentially performing the same kind of compression that gzip does, but in a significantly less efficient and more complicated manner.
We should be gzipping API payloads.

Wombat

Toby double quotes around keys

double quotes around keys

It should be around property names as well, not just keys. That's most likely why.

Basically, this is how I got my data:

I agree with the gzip thing will circumvent the reason to do alphanumeric naming though, but I think you should give it another try with removing the quotes from property names, not just keys.

Regardless of a GZIP compression, it's pure data that is being saved (those quotation marks are not being piped across!)

Toby

Unless I'm missing something completely... property names are keys! Any string surrounded by two quotation marks and immediately followed by a colon, as my regex shows.

That's the way I got my data too.

Yes, it is technically saving some data, but when you gzip it is so negligible that it's not worth the trouble. Read about how gzip/compression works. Because property names/keys are so repetitive, it will factor the quotes out so they will barely add a few dozen bytes to the compressed size – as I have shown.

Wombat

Toby Hmm, I always thought keys are different than property names:

0: {
propertyname: 'Toby',
propertyname2: 'Muffin'
}

0 is a key that is usually in quotes, and propertyname, and propertyname2 are.. propertynames but usually in quotes. So that's why I got a bit confused when you only mentioned keys because they are property's that are in quotes as well, not just keys? (Keys in this case are too).

In any event, if your data was object literal I believe it's negligible I guess. Oh well, worth a shot, I guess I forgot about gzip compression.

Edit: Because from what I see, I would agree with you, it's negligible / not worth it when gzipping it.

My test was just conducive to regular plain text, that's why I think.

Toby

Wombat Glad we've reached an understanding. I've enabled gzip on API payloads now 🙂