Show HN: Concise Encoding – a friendly data format for human and machine

by kstenerud • 6 years ago

78 46 6 years ago

dfischer• 6 years ago

Hmm seems like a battery packed version of bencoding but maybe someone can enlighten further?

kstenerud18• 6 years ago

Although it uses text values for delimiters, Bencode isn't designed to be edited in text mode. If you want to transmit binary data that can be efficiently read by a machine, you cannot also make it human readable/editable - they have opposing goals. The only way to get both is to have a twin format (binary and text) that is 1:1 compatible and transparently convertible.

IshKebab• 6 years ago

Native support for complex types like URLs and UUIDs is a mistake. It just makes it more complicated and harder to interoperate with other formats like JSON. Look at all the archaic types ASN.1 supports.

kstenerud18• 6 years ago

URIs and UUIDs are never going away. We're always going to need a way to lookup resources, and we're always going to need unique identifiers. To not include them is a mistake, because you then have to add a (potentially incompatible) extra layer of encoding at the application level just to store them in the encoding format.

ysleepy• 6 years ago

URL parsing is a nightmare. There are lots of RFCs, but the real world is full of quirks. This will lead to edge case differences between the implementations.

Also, which URI standard is part of the spec, what if the URI standard evolves? – It is a complex standard with multiple RFCs and a long history.

What of possible, different length restrictions in language native URI types?

I have no stake in this though, feel free to ignore. I tend to see negatives first.

Aeolun• 6 years ago

Personally I think it’s nice, though I think they’re basically syntactic sugar for a string and byte array.

Does the format validate that a URL is actually a URL?

kstenerud18• 6 years ago

There's nothing stopping someone from putting a URL validator into their codec, but it's probably a lot less work to just pass it to your runtime library's URL parser.

tasogare• 6 years ago

I disagree. Just because a common data format doesn’t support those types doesn’t mean other formats have to do the same mistake (and given that JSON lacks comments and an official formal spec, its design is questionable to begin with). If interoperability is needed with JSON or other formats not supporting date or uuid, such data type can be serialized as strings.

jiehong• 6 years ago

CBOR is filling the same requirements and is isomorphic to JSON.

kstenerud18• 6 years ago

Yes, and because of that it suffers the same downsides of JSON. You also can't just transparently convert CBOR to JSON because of the data type disparity.

x3ro• 6 years ago

I think it’s interesting that they included a markup type. Based on the spec it’s „similar to XML“, so I suppose this markup is of their own making.

It doesn’t seem like this would make transmitting XML or HTML significantly more efficient to transmit compared to just gzipping a string. And if you wanted to use this, you’d have to write the conversion first.

Can someone help me understand the use case for this, ie in which situations this would be super useful?

kstenerud18• 6 years ago

The idea is to allow presentation data to be included inside of general data, and also to allow presentation data to contain any kind of general data (not just string keys and string values). At some point in the distant future, it would also be nice to have official numeric tag assignments in schemas (e.g. 1 = "View", 2 = "TextBox", etc) to further improve processing by eliminating the need for text parsing just to decide what kind of object to present. These would be transparently converted from the text-based names in the human-readable text format when converting to the machine-read binary format.

The overarching purpose of Concise Encoding is to bring forth the tools to make transmitted data accessible to humans, and at the same time efficient to process by machines.

kstenerud18• 6 years ago

If you have ideas or comments on the design of the format, please let me know! I've been working on this for 3 years and am getting close to release, but there are always blind spots when working alone.

The basic premise is that you can never marry "easy to edit" with "efficient to process". They pull in opposing directions, so the only alternative is to have two 100% compatible formats: one in text and one in binary.

I've chosen the data types carefully to support the most common types that come up in real world situations (and have included custom type support for data not intended for public consumption). The compromises made should allow most people to work without adding extra encoding on top of the encoding (base64, special text parsing and such).

The reference implementation is working for all common cases, and mostly there for array types. Once that's finished, I'll start on the schema design.

bb88• 6 years ago

You need a fixed decimal float to represent currency.

2.25 or 1.10

kstenerud18• 6 years ago

The decimal float type would handle that, with a schema to specify fractional precision.

mkesper• 6 years ago

Would readable large integers be possible? E.gPython. gathered support for numbers like 1_000_000_000

kstenerud18• 6 years ago

Absolutely. Numeric data can be arbitrarily large, and supports _ as numeric whitespace in the text format.

valw• 6 years ago

Are you aware of EDN, Transit or Fressian, which IMO improve a lot on the currently common approaches to encoding (JSON / YAML)? If so, how does Concise Encoding differentiate itself?

https://github.com/edn-format/edn

https://github.com/cognitect/transit-format

https://github.com/Datomic/fressian/wiki

kstenerud18• 6 years ago

Yes, I've looked at EDN before. It's solving a similar problem, but not the same one. EDN is focused on extensibility, where as Concise Encoding is focused on concision, and binary-text compatibility.

Transit is a bridging format to other encoding formats, which is very cool, but a fairly different problem space.

Fressian is pretty close, but is binary only. The main purpose of Concise Encoding is binary-text compatible formats.

carapace• 6 years ago

In a project like this I look for a "related work" section (that shows you know the context and are at least not reinventing [square] wheels and have some reason why your thing is different/better than all the other things that do what your thing does already) and a formal grammar (that I can feed to a parser generator etc.).

seanalltogether• 6 years ago

It's weird that you use a completely different annotation style for byte arrays. Why not just use

bytes = u8[10 ff 38 9a dd 00 4f 4f 91]

9214• 6 years ago

The basic premise (The friendly data format for human & machine) instantly reminded me of Red language [1], which, by the virtue of homoiconicity, is also its own data format (both textual and binary [2]) with 50 datatypes, many with several unique literal forms:

  date:      4-Oct-2020 4/11/20
  time:      13:04:07
  url:       https://concise-encoding.org
  email:     kstenerud@gmail.com
  ref:       @kstenerud
  tag:       <div color="red">
  integer:   123 
  float:     123.45 1.#NaN -1.#INF
  pair:      67x89
  percent:   9000%
  money:     USD$0.50
  binary:    #{deadbeef} 2#{00101101}
  container: [#hashtag "string" %file.ext #(key value)]
  ...

[1]: https://www.red-lang.org/p/about.html

[2] https://github.com/red/docs/blob/master/en/redbin.adoc

spfzero• 6 years ago

Really interesting, thanks for posting this. Edited to add: interesting for the cross-platform and independence aspects.

viraptor2• 6 years ago

So while I like the idea of asn-like multiple encoding standards, there are some fragments in the description that make me just see potential issues:

Support for the NUL character (u+0000) is implementation defined due to potential issues with null string delimiters in some languages and platforms. It should be avoided in general for safety and portability. Support for NUL must not be assumed.

I believe it's easier to force awkward languages to support nul character than deal with the chaos of not knowing if it's supported or not. They're are other places where they say "this is hard, we'll let implementations to not support it" and it's a ticking time bomb.

Also it doesn't look like you can deduplicate map keys:

A reference must not be used as a map key.

Which means as much repetition and space waste as in json.

kstenerud18• 6 years ago

I'm still on the fence with NUL to be honest... I may remove it, but for now I want to garner discussion.

I'm not sure how disallowing a reference for a map key would cause repetition and space wasting. Keys are generally small, so a reference would in many cases end up taking a similar amount of space. Can you talk more about allowing references as keys? I need different points of view NOW, before I finalize v1 of the format! The negative is that it complicates processing, since you could reference a non-keyable value and attempt to use it in a key.

Everybody PLEASE chime in with your thoughts on the format, because I want to make sure I get it as close to right as I can before I finalize V1!

pdimitar• 6 years ago

As a programmer who had to wrestle with like 30 formats during his career, here's one piece of non-negotiable requirement:

---------- NO. OPTIONAL. PARTS. OF. THE. STANDARD! (And no "but"-s.) ----------

Start there. This might sound flippant and snarky but I promise you that it isn't. Don't ever leave anything to the implementations.

Oh, and build your own compliance tool. So people can just run the results of their own implementations against your tool and see if their code is compliant with your data format or not.

grncdr• 6 years ago

Not the GP but regarding no references for map keys; I can imagine that this would be an issue when representing a list of maps with uniform keys. E.g. a list of 1000 points with the keys “latitude” and “longitude”.

I can think of various more compact ways of structuring the same data, but JSON + gzip encodes the naive structure in a very compact way with very little programmer effort.

In general, the design seems to miss a compact representation of an array of homogeneous records, which is a common use case IME.

Edit: Also I agree with GP regarding NUL and other comments about no optional stuff.

sitkack• 6 years ago

Optionality is a bigger problem than people realize, these special cases force “conditional pollution”, and when the optionality is implemented with exceptions, you now have to understand non-local control flow.

Make everyone’s lives easier and make a spec where everything is mandatory.

ethbr0• 6 years ago

No spec writer is omniscient.

I think the best way to deal with this is explicit forking / special-purpose subspec. Where again, in that spec, things are different / extended, but still mandatory. And if consumers fall into that usecase, they can explicitly advertise / consume that version.

spacemanmatt• 6 years ago

How does this address format creep? Why is it going to be better to support all the existing formats and now this one too? Does it really add that much?

The first formats filled a void. This format joins many players already on the field.

kstenerud18• 6 years ago

It's versioned, so if we come up with significant improvements later, we can bump the version.

This format fills two voids:

1. The lack of native types in most of the other formats

2. The lack of either editability or processing efficiency, depending on whether it's a binary or text format.

I developed this format because I'm tired of putting encodings on top of encodings (i.e. base64) and other such tricks, just to get my data across to the other side, and I want a general purpose data format that's efficient as well as readable.

mpweiher• 6 years ago

Glad to see more work in this space.

You might spend a little more saying what makes your approach unique:

"Use text based formats that are bloated and slow, or use binary formats that humans can't read. Wouldn't it be nice to have the benefits of both, and none of the drawbacks?"

NeXT/Apple property lists have had that feature for decades now, with multiple text and binary serialization formats.[1]

Infra [2] also makes very similar claims "Existing metaformats fit neatly into two categories. They are either textual for human-readability (such as XML and JSON) or binary for compact serialization (such as Thrift and Protocol Buffers). Infra can play the role of either, imbuing each with the desirable properties of the other."

[1] https://en.wikipedia.org/wiki/Property_list

[2] http://www.christopherkhall.com/research.html

kstenerud18• 6 years ago

Property lists came very close, but still suffer from a number of issues:

- The binary format is not efficiently packed. For example, Concise Encoding uses 200 of the 256 codes to directly encode integers (-100 to 100) since they are the most common data values in the wild (along with short strings, which also have their own special encoding).

- There's no array type (as in contiguous objects of the same type and size)

- Dates are in seconds, not Gregorian fields, and have no time zones.

- Binary data in the text format is base64 encoded, ensuring that it's 100% unreadable in its raw form.

- Container types are prepended with a length field, making progressive construction impossible.

Infra looks interesting, but it strays from editable text into a kind of hybrid format that requires a specialized editor to read and write. This would only work if such editors became ubiquitous, which is unlikely. I ran down this path for awhile as well, but finally decided against it.

wsargent• 6 years ago

How does this compare to Amazon Ion?

https://amzn.github.io/ion-docs/

kstenerud18• 6 years ago

Ion almost got it right. The problems are:

- Binary format is big endian. All modern processors use little endian.

- The time format doesn't have a time zone.

- The binary time format is in UTC, while the text format is in local time, and you have to convert (WTF!)

- The binary time format is HUGE.

- Typed nulls seems a little excessive for very little gain.

- No efficient small integer encoding.

- Containers in the binary format are prepended by the element count, which makes progressive container filling impossible.

- No custom types (except maybe s-exps, which are overkill)

- No metadata

- No URI, UUID types

- no markup container

- no references

evnix• 6 years ago

how do you do multi line strings, the thing I don't like about YAML is there are 101 ways to do a multiline string.

kstenerud18• 6 years ago

Two ways: Quoted string using backslash-newline (like in C), or using verbatim strings (like here documents in bash).

formerly_proven• 6 years ago

I really like the inclusion of hexfloats (beware of C++ iostream bugs, though), because they're wayyyyy easier to parse correctly than decimal floats (way faster, too). As an added bonus they're reasonably easy to read for developers, too, since they're just scientific notation in hex.

08-15• 6 years ago

Seriously, another one? https://xkcd.com/927/

Why was CBOR not good enough this time? Still not enough string types? Oh, indeed, CBOR doesn't have markup, whatever that means.

Show HN: Concise Encoding – a friendly data format for human and machine

Related Stories