Concise Encoding: A secure data format for a modern world

by bshanks • 4 years ago

76 42 4 years ago

kstenerud7• 4 years ago

Hey guys, sorry for the slow updates on this. Ukrainian issues have swallowed up most of my time recently, and my OSS projects have suffered as a result.

Rest assured that this project IS an ongoing concern, and is the foundation of many more technologies I intend to bring to bear over the coming years.

samatman5• 4 years ago

I'm glad to see this surface again, I lost track of it for a couple years and surfaced it about six months ago.

Are you open to feedback in the spirit of collaboration? I've considered opening an issue or two on the repo, but it always feels a bit presumptuous to do that when one person is the author of the entire project.

I like the way you think about these things and I find CE quite promising.

kstenerud7• 4 years ago

I'm absolutely open to feedback, criticism, collaboration of any kind! The whole point of a format is to be usable, and while I have a number of opinions on the subject, I'm no superman. I may not always agree with other peoples' opinions, but even when I don't, the whole point is the conversation it sparks; that's the nexus of greatness.

One of the main reasons why I haven't released version 1 yet is because it's been so much of a one-man-show that I'm under no illusions of being immune to tunnel vision.

In the immortal words of Johnny 5: I need input!

munro• 4 years ago

Looks really cool--I love the overview, we've definitely grown out of JSON, most of the time I sadly end up using pickle when using Python.

Too many question marks in my head to be honest. The more I think about it, I believe code still may be the the best way to describe really complex data.

What is interesting to me is the holistic view of all the data types we most commonly use--but then I don't really know how this differs from Thrift/Avro/etc--and then further, why haven't we as a community moved to one of those?

manholio• 4 years ago

This is pretty cool, definitely would have come handy 15 - 20 years ago when people like Cohen and Nakamoto were inventing Bittorrent and Bitcoin, rolling their own binary serialization atrocities and unleashing them on humanity.

FridgeSeal• 4 years ago

You mean you don’t like swapping between endianness 15 million times when you’re processing Bitcoin????

BoppreH5• 4 years ago

Things I liked:

- Versioning.

- Time zone identifier instead of just a fixed offset (which are ambiguous for future events).

- Native encoding of binary values.

- Graph notation with support for labels.

- Comments!

- Trying to escape lookalike characters, even though I think that's a lost cause.

Things I'm not so keen about:

- NUL character in strings being platform and settings dependent.

- Line break not being forced to a consistent value.

- The most complicated number encoding scheme I've ever seen (e.g. 0xa,3fb8p+42).

- Entity references are a footgun for anyone writing a depth-first or breadth-first algorithm.

- Arrays-vs-list feels like it doesn't belong in encoding formats.

profquail• 4 years ago

The complicated number encoding scheme you mentioned is a hexfloat: C has them too.

Hexfloat can be really useful when you need precise/exact floating-point constants for numerical methods. Without them, you end up having to do more-complicated hacks to preserve exact constant values when code gets compiled, or you have to live with compilers (sometimes) subtly altering constants.

I wish more languages supported hexfloats.

BoppreH5• 4 years ago

Thanks, I had never seen that before. But Concise Encoding did complicate them further by accepting commas as decimal separator.

unwind• 4 years ago

Just to clarify a bit what "C has them" means; the standard library string-to-float parsing function strtof() [1] support this format.

You can also use them as floating-point literals in your code.

[1] https://linux.die.net/man/3/strtof

formerly_proven• 4 years ago

Hexfloat is also vastly easier to parse from/to IEEE 752 format than decimal floats.

lifthrasiir4• 4 years ago

Time zone identifier instead of just a fixed offset (which are ambiguous for future events)

Time zone identifiers are not future-proof. As a concrete example imagine a fictitious country X with the largest city (not necessarily the capital) being Foo and a single time zone throughout the country. A natural time zone identifier would be something like `Region/Foo`. If a part of X switches to another time zone and Bar is the largest city in that region for whatever reason, there would be another time zone identifier `Region/Bar` assigned. You can't automatically determine which `Region/Foo` should be converted to `Region/Bar`. It is up to you to pick a strategy in such cases, but time zone identifiers themselves do not solve the ambiguity problem.

Besides from this false sense of security, time zone identifiers were not meant to be for general interchange and there are several understandable but problematic assignments. Baking them into a data structure format doesn't sound a good idea.

BoppreH5• 4 years ago

That's a valid scenario where time zone identifiers don't work well, but what's the alternative? A fixed offset is guaranteed to be wrong on DST switch, and that happens a lot more often than time zones splintering. I guess GPS coordinates would be more accurate, but then you have privacy concerns.

Fun fact: time zones are named after cities for robustness:

  Country names are not used in this scheme, primarily because they would not be robust, owing to frequent political and boundary changes. The names of large cities tend to be more permanent. Usually the most populous city in a region is chosen to represent the entire time zone, although another city may be selected if it is more widely known, and another location, including a location other than a city, may be used if it results in a less ambiguous name.

https://en.wikipedia.org/wiki/Tz_database#Location

avidphantasm• 4 years ago

It seems like these problems with defining/describing timezones (i.e., temporal reference systems if you will) are similar to those of defining spatial reference systems. Spatial reference systems are fairly rigorously defined/described using both standard (e.g., WKT) and non-standard formats (PROJ strings, EPSG numbers). Is there something similar for temporal reference systems?

devonkim• 4 years ago

Time zones based upon anthropocentric concepts like countries have issues simply due to the realities of political changes. Entire countries and civilizations can come and go. The concept of even “common era” is a convention of convenience still based around a religious figure frankly, which is a sort of retcon given others through the same time period were likely measuring time in another coordinate system essentially.

A self defining and self describing set of measurements would help here but obviously bloats formats similar to how XML schemas become overly verbose and pedantic without adding clarity to the target audience.

So I agree that baking in a time zone is likely a bad idea but I have doubts we can do much better by trying to use another more absolute time coordinate system independent upon human civilization conventions.

baq2• 4 years ago

'simple' solution - time zones should have the time zone DB version included. should be future proof, too.

samatman5• 4 years ago

Re. time zones, I'm of the increasing conviction that they should be avoided whenever possible, and it usually is.

I think of a time zone as drawing a large box on a map, for any given instant in time. Future events either happen 'only in time' or they happen at a time and place. A time zone isn't a great representation of a place! I'll be much happier storing UTC and coordinates, then turning that into a timezone for display when I have to.

A time zone is an approximate (and weird) spacial coordinate system which causes nonlinearities in the representation of time, and unless you want both of those properties it comes with baggage.

sparsely• 4 years ago

A time zone is an approximate (and weird) spacial coordinate system which causes nonlinearities in the representation of time, and unless you want both of those properties it comes with baggage.

That approximate and weird system is how humans tend to think about future events though. You're right that often it is not necessary, but for many usecases (e.g. when a TV show will be first broadcast in the future) they are the most robust model we have.

kstenerud7• 4 years ago

NUL is a tricky beast even under ideal conditions. In the first draft I'd simply forbidden them. Simple, decisive, and of course people didn't like that. then I did allow it, and suddenly the security gates were flung open. Some platforms simply cannot support nuls in strings, so now it becomes platform dependent (bad from a security perspective). So then my next attack vector was to make it configurable with a default of deny. Better from a security perspective, but now it's complicated. TBH there are nights where I think I should just go back to no NULs at all again. But I really need more discussion with stakeholders to figure out what problems this solves and how important it really is...

Line break is another sticky issue. Non-technical Windows users will inevitably produce documents that use CRLF, so if it rejects the file, that's a bad user experience. What's the best trade-off here? I'm not really sure.

The number encoding thing is being discussed in other comments so I'll leave it be. I had allowed comma because that's how a huge chunk of the world represents the decimal separator. Once again, this is about user experience with the non-technical users. I'm REEEEEEEALLY on the fence with this one.

Entity references are dangerous, yes, but also powerful. The point of them in the format is to solve the recursive reference problem, because you just can't do that otherwise, and these structures do exist in the world. It's another case of an imperfect solution for an imperfect world. Bear in mind I absolutely do NOT want this format to become some Turing complete language. This is just the minimal feature set I could think of to represent real world data.

Arrays-vs-list is another one of those compromises. Encoding an actual array of fixed types into a list would be slow and bulky, leading people to just encode them as a chunk of bytes like they currently do in JSON and other formats. As an imperfect solution to an imperfect world, I want to at least let people preserve the semantic meaning of what they're sending since they're going to use array encodings regardless of what the format supports.

BoppreH5• 4 years ago

Thanks for your reply! It's a really good proposal, and I'll definitely consider it next time I'm serializing something.

NUL is a tricky beast

One alternative I found interesting is Modified UTF-8 (https://en.wikipedia.org/wiki/UTF-8#Modified_UTF-8), where NUL is encoded as 0xC0 0x80. It's already in use in Java, apparently. Disclaimer: I've never heard about it before, and I don't know how well supported it is.

Line breaks and commas

I'm from a country that uses comma as decimal separator, and I still prefer dots when programming. I dread ambiguous numbers like "1,001". I'm confident this is true for almost all technical people. And I really don't see non-technical users editing this kind of file.

If they can be trusted to read it, and to modify it without introducing syntax errors, they can be trusted to use dots and an editor that shows LF as linebreaks (i.e. anything but `notepad`).

My choice would be dots only for decimal separator, accept LF and CR LF on reading, but prefer LF when writing.

Entity references

You can always define manual ids:

    c1
    {
        "alice"={friends=["bob" "charlie"]}
        "bob"={friends=["alice"]}
        "charlie"={friends=["alice"]}
    }

Of all data types that could have native encodings (colors, IP addresses, lat/lon coordinates, enums, tags, markdown, hashes, file permissions, OIDs, DOIs, ISBNs, etc), I think arbitrary object graphs bring too many downsides for a serialization format.

You don't want reader CVE's because of Billion Laughs, and you don't want to force every programmer writing a simple traversal algorithm to correctly handle cycles.

marcosdumay• 4 years ago

Hum... For nul, it's common to have an escaping sequence (yes, on binary data) and use it to encode the problematic characters. It's for the best if you encode enough of the data for one to be able to dump your file into a terminal and nothing getting compromised on the way (the terminal just failing to work is ok).

Personally, I disagree with how your format handles all those other issues too (except for the numbers), but well, if you think you are correct, go try it. If it works, it works, and my disagreement may easily be misguided. Anyway, I disagree because:

For the line breaks, the internet has a way of trying to "fix" them and completely breaking the line-information of the original document. It would be ok if the format wasn't blank-space dependent, but it is, so changing the lines breaks the data. Anyway, that is becoming a lesser problem with time, so maybe for a new format it's fine.

Entity references on formats that are not focused on them are surprising. That means a lot of software will break once they get one, and tradition says they will do that in a way that compromises computer security. I would either change the format so that references are almost always used or remove them. If an application needs references, it can always tag the entities with an id and put the references there by itself.

The same applies for arrays, in a lesser degree. They will be surprising, but they are also easier to handle. But they are also much less necessary, since lists can always replace them. I'm really not sure if they are a net negative or positive.

baq2• 4 years ago

IMHO schema support should be part of the initial release. https://json-schema.org/ being an afterthought to the original JSON was a mistake.

kstenerud7• 4 years ago

Yup, that's the idea. I've been looking at CUE for this

https://cuelang.org/

barnabee• 4 years ago

Would be nice to support ISO8601 dates and times rather than (or in addition to) defining a similar custom format IMO

davibu• 4 years ago

Many data set are stored like that (Json separated by RC):

{"name":"name1", "phone":"+769989823"}

{"name":"name2", "phone":"+769244563"}

{"name":"name3", "phone":"+769989295"}

...

There should be a way to do the same thing with concise.

lifthrasiir4• 4 years ago

Not Concise, but CBOR does have a proposal for such data [1]. This is implemented via CBOR's tagging system where you can attach a tag to otherwise mundane data to hint otherwise, which is in my opinion one of the biggest idea behind CBOR and why CBOR is not really a ripoff of MessagePack.

[1] https://github.com/kriszyp/cbor-records

sgammon3• 4 years ago

thanks i hate it

sgammon3• 4 years ago

we do not need special types for UUIDs. why are we doing this to ourselves

sgammon3• 4 years ago

Times are different from the carefree days that brought us XML and JSON.

no they are not lol

Concise Encoding: A secure data format for a modern world

Related Stories