JCOF: JSON-like Compact Object Format

by todsacerdoti • 4 years ago

35 68 4 years ago

faleidel• 4 years ago

I would like to see gzip compression added to the benchmark

mg• 4 years ago

    curl https://raw.githubusercontent.com/mortie/jcof/main/tests/corpus/meteorites.json | gzip -9 | wc

    Gives me 34569

So the comparison is:

    JSON: 244920 bytes
    JCOF: 87028 bytes
    GZIP: 34569 bytes

eof• 4 years ago

To be fair you would gzip the JCOF encoding in this example too.

Author mentions gzip doesn’t work for some use cases. For use case mentioned I’d expect sqlite to be similar, at least that is the default thing I’d reach for.

If for some reason sqlite wasn’t sufficient probably a custom binary encoding controlled and updated via code instead of config would be next.

KerrAvon• 4 years ago

Serialization formats that dedupe and perform other fancy processing trade increased CPU time for decreased memory usage. People who care about CPU time will want to measure the CPU time hit as well as the memory savings.

This format is also not interoperable without a decoding library, which kind of invalidates the comparison to JSON. If you're going to do this, why not just go full binary and save even more memory?

throwawaymaths• 4 years ago

You're decoding JSON in all cases. Usually network dominates over cpu. Sometimes it's useful to have human readable serializations, especially if you expect to operate in anger.

lijogdfljk• 4 years ago

Depends on your needs. When i'm really concerned about deserialization time, i try to skip it entirely. Partial|Total zero copy deserialization is the thing to reach for. Various libs exist, https://rkyv.org/ is what i've been toying with recently.

For those unfamiliar, here is the interesting bit:

rkyv implements total zero-copy deserialization, which guarantees that no data is copied during deserialization and no work is done to deserialize data. It achieves this by structuring its encoded representation so that it is the same as the in-memory representation of the source type.

linker3000• 4 years ago

Slight awkwardness with the spoken pronunciation.

taddevries• 4 years ago

Glad I'm not the only one who reads it that way.

jsiaajdsdaa3• 4 years ago

J cough?

sto_hristo• 4 years ago

Thing about JSON is the ease of reading and writing it, regardless the size of the document. Compression deals with the rest.

Someone• 4 years ago

It may be easy to read and write, but few libraries implement it to the letter because the format doesn’t support extremely common use cases.

For those who disagree, try finding a json library that can reliably round-trip the data in a json file (i.e. ignoring spacing, tabs, and extra line breaks)

Common breaking points are discrimination between integers and doubles (e.g read 0.0 and write 0) and limits to the number of digits in a number.

It’s futile to try and change it now, but I think json would have been better if it had disjunct sets of integers and floats, did 64-bit ints (1) and IEEE doubles (1) as opposed to arbitrary length integers and floats, standardized serialization of time stamps and durations.

Requiring parsers to keep keys in file order also might have been a good thing.

(1) I know that opens a rat’s nest of users wanting unsigned 64-bit integers, signed 16-bit, 128-bit quad floats, etc, but I don’t see how not addressing that problem at all is better than picking reasonable values, event though that gives up on supporting those alternatives. Now, we have libraries making choices there, hopefullly in compatible ways.

HelloNurse3• 4 years ago

Neither fully human-editable nor really compact, but an interesting experiment nonetheless.

andrew_• 4 years ago

Nevermind readable

mywittyname• 4 years ago

I'm still not sure wtf the leading 0, on every object is supposed to represent.

bearjaws2• 4 years ago

People keep reinventing JSON trying to beat gzip, and it never beats the simplicity of gzipping JSON...

Beltiras7• 4 years ago

This actually does. I used Python gzip to see how the minified would compress and came up with 157 bytes to JCOF's 134. Using the same mechanism to gzip the JCOF saves only 4 bytes. I'm actually rather impressed.

tmp_anon_22• 4 years ago

I think people would reject your samples because the data size is too small. Production JSON comes in all shapes and sizes and situations where you care about performance usually indicates much larger payloads.

zimpenfish7• 4 years ago

It might on small files but it doesn't on (some) bigger ones. e.g. I dumped a Minecraft region file into JSON giving 85M. `jcof` turns this into 66M. `gzip` on the original json gives 14M for `-1` and 11M for `-9`. `zstd` also gives 11M.

But because the Minecraft JSON is highly redundant, I also tried it on 84M of fake social graph data. `jcof` gave 44M, `gzip -9` gave 19M and `zstd` gave 18M.

In summary, neat idea, but you're maybe better off with `gzip` or `zstd` in the real world.

xsmasher2• 4 years ago

comets.json:

        JSON: 51949 bytes
        JCOF: 37480 bytes (0.721x)
  JSON ZIPED: 15178 bytes

Zipped json wins again.

conradev3• 4 years ago

Now I am curious – how does zipped JCOF turn out? I know that’s not the author’s intention, but I’m curious how it compresses

heresie-dabord• 4 years ago

People keep trying to re-invent a human-readable, parsable, efficient, robust format that can be validated.

But "easy" wins so we keep returning to CSV and JSON. ^_^

nine_k2• 4 years ago

Gzip may be standard and supported transparently by your http library.

OTOH zstd [1] should be significantly more efficient, while being as fast or faster.

[1] https://en.wikipedia.org/wiki/Zstd

indymike3• 4 years ago

JCOF is not usable in many of the use cases where JSON is. Here is why:

* JSON is really schemaless so I don't have to assume all objects are shaped the same or that they are even the same kind of object. This allows for streaming serialization, and does not require the data structure to be known or to introspect data to create the heading lines.

* Nested objects look to be difficult, especially if the schema is not known.

* JSON is very human friendly, JCOF is not.

I do think the format is better than CSV, TSV and other delimited flat record formats. JCOF appears to be able to handle objects that are not tabular, which CSV just can't really do. A replacement for JSON this is not, and storage efficiency isn't really relevant because of the difference in purpose.

nine_k2• 4 years ago

JCOF is just not radical enough.

It should stop pretending to remain human-readable, go binary, and become a variant of protocol buffers, thrift, etc.

BTW being schemaless is a boon during initial hacking things together, and an impediment when operating and developing the system further down the line. It's the data equivalent of dynamic typing vs static typing, interpreted vs compiled in code.

indymike3• 4 years ago

It should stop pretending to remain human-readable, go binary, and become a variant of protocol buffers, thrift, etc.

There are lots of ways standard ways to do serialization to binary formats. https://en.wikipedia.org/wiki/Comparison_of_data-serializati...

BTW being schemaless is a boon during initial hacking things together, and an impediment when operating and developing the system further down the line. It's the data equivalent of dynamic typing vs static typing, interpreted vs compiled in code.

Sounds like XML, which is a whole other discussion :-)

mort962• 4 years ago

Hey, JCOF author here. I didn't notice this had made it to HN before now.

* JCOF allows for streaming serialization if you want, you can just use the key-value object syntax and use inline strings everywhere. You won't get much size improvement over JSON when using a streaming approach, but you can even do a half-and-half approach; maybe you have a set of strings which you know you're using a lot, so you put those strings in the string table, then you do a streaming encode of the rest of your data, with string references to the string table. You could even do the same with object shapes, where you hard-code a couple of object shapes you know you use a lot, and use the JSON-style key-value syntax for other kinds of objects. Basically, you can choose where you want to be on the efficiency vs streaming curve.

* Nested objects aren't difficult; an object can go in any place where a value can go, just like in JSON. This object can be either a "shaped" object where the keys are listed in the object shape table, or it can be a "keyed" object where you write out the keys inline (as either literal strings or references to the string table).

* JCOF is human-writable; you can write a document which looks a whole lot like JSON. This is a valid document: `;;{"people":[{"name":"Bob",Aage":36},{"name":"Alice","age":40}]}`. But optimal JCOF is mostly unreadable. So it's less human friendly than JSON, more human friendly than binary formats; it's a trade-off.

I think JCOF hits enough interesting, unexplored points on enough relevant curves that it has some value. It also ends up being smaller than any of the binary formats I've tried like MessagePack and CBOR (even with the string references extension), while being text-based. It's obviously not going to replace JSON, but it's useful in some cases.

indymike3• 4 years ago

Oh, I think you have something, but I don't think JSON and JCOF are comparable. I think it makes a very interesting replacement for formats where there is a header line like CSV, TSV. JCOF allows for some level of consistent nested structures (which would be nice for header/detail records which are really common in the real world) and would be so much nicer than many of the bespoke formats you see out there... and also, sooo much more efficient than XML, and without the issues that come with XML. Incidentally, these kinds of uses are where JSON really doesn't work well because you always have the potential of a mis-shaped object.

In short: nice work. Might be fun to roll an implementation or two :-)

Scaevolus• 4 years ago

gzipping stuff from JavaScript in the browser is inconvenient until TextEncoderStream is supported in Firefox

There are a pure-JS DEFLATE implementations with reasonable performance-- I recommend fflate.

peanut_worm• 4 years ago

JSON is supposed to be human readable this doesn’t even seem like it would address the same issues.

m463• 4 years ago

I'm always conflicted.

They're improvements, they're experiments, they're an attempt to move forward. Somebody cares!

but Randall Munroe said it pretty well: https://xkcd.com/927/

talideon• 4 years ago

JSON isn't mean to be efficient, but to be a good compromise between being human readable/writable, easily generated/parsed by a machine, and familiar. CBOR and BSON make more sense, to be frank, for any use case I could think of using this in.

xsmasher2• 4 years ago

I don't think BSON de-dupes the keys in the manner that JCOF does? But it has other nice properties. In a game I worked on we used a BSON library that didn't really "parse" the data when you loaded a file; it just kept a pointer to the original binary data and let you perform queries against it.

We used this to keep our memory footprint low; instead of loading BSON files we would memmap them (getting a pointer that can be used to read the data right off the disk) and passed that to the BSON library. Bang, we could map a 10MB file and it wouldn't use 10MB of system memory; it would just swap portions in and out as they were used.

This has performance tradeoffs but for us (on mobile, where there is no swap space) the memory savings was worth it.

dathinab• 4 years ago

no offense, but this seems pointless tbh.

If I need a more compact no longer at all human readable format I can use a compact binary format. If not I can use JSON which has wide spread support.

or just use a compression (after which the difference between JSON and JCOF should be negligible)

mort962• 4 years ago

I'm curious, which more compact format would you use? Because I have looked a bit and been unable to find one, both MessgePack and CBOR produce pretty large objects. CBOR with the string reference extension comes closer, but it's still bigger, and string references don't seem very well supported in the CBOR ecosystem.

Do you mean you'd use a format with a schema?

LAC-Tech• 4 years ago

The comments here are really negative.

I think this is a great idea. JSON is incredibly wasteful and any improvement is welcome.

And yes, you can gzip JSON. still wastes a lot of space when you actually need it in a format where you might want to read individual fields.

smitty1e• 4 years ago

If a specific use-case can benefit from it, fine.

I love me some jq, not moving much data as JSON, and so this amounts to an interesting exercise.

JCOF: JSON-like Compact Object Format

Related Stories