Rows of String Values (RSV Data Format) Specification

https://github.com/Stenway/RSV-Specification

by greggyb • 3 years ago

2 2 3 years ago

PaulHoule• 3 years ago

Note there already are official characters to use for delimiting fields and records.

https://en.wikipedia.org/wiki/C0_and_C1_control_codes#Field_...

and if you use ESC an escape character then the special codes are all in sequence like they are in that spec. For that matter you know that none of these characters appear in ordinary text (the world forgot about FS, GS, RS, US) so if your strings are UTF-8 encoded text you never have to use ESC.

But really if you are going to binary formats and your data is mostly numeric there are strong reasons to go to a columnar format like

https://parquet.apache.org/

although this is incompatible with the "ragged" data structures that this spec supports.

greggyb• 3 years ago

I don't think I would ever consider RSV and Parquet alongside each other. I also don't think I've ever personally had a use case where the proposed RSV format would have been a good fit. Nevertheless I found it interesting.

Even if you were to use FS, GS, RS, US, they are all parts of valid UTF-8 strings. I agree that they would be vanishingly rare in most corpi of "ordinary text". The author here is also aware of them.[0]

I found it quite an interesting idea to use bytes that are invalid in UTF-8 strings as delimiters for data that is guaranteed to be encoded as UTF-8 strings. This moves those control characters properly out of band. Out of band is a much stronger guarantee than vanishingly rare.

I don't think I'll use RSV anytime soon, but I appreciate the creativity that went into it, as well as the many implementations the author has already collected[1]. There are 31, by my count, including several that support larger ecosystems: C (everything can call into C), C# (dotnet runtime and other languages, such as F#), Java (JVM languages, such as Clojure), JavaScript (any of the many compile-to-JavaScript langs, such as TypeScript).

[0] https://github.com/Stenway/RSV-Specification#why-dont-you-us...

[1] https://github.com/Stenway/RSV-Specification#how-will-rsv-ga...

Rows of String Values (RSV Data Format) Specification

Related Stories