No JSON/MAPS interoperability in 17.0?

http://erlang.org/pipermail/erlang-questions/2014-March/078228.html

by rahij • 12 years ago

44 82 12 years ago

erlang.org

EC12• 12 years ago

I do not have a magical format to propose to replace it

So why did you even comment in the first place?

mikeryan• 12 years ago

Seriously, he just lost his own argument right there.

mahmoudhossam• 12 years ago

He did say he uses msgpack which he claims is a step in the right direction, although it too has its issues.

garretraziel• 12 years ago

I don't agree with you. I think that someone can point out problem without providing the solution at the same time.

barbs• 12 years ago

He didn't just point out a problem, he actively tells people to "stop using JSON" without providing an alternative.

HugoDias2• 12 years ago

But when he said "Stop using JSON" he needs to provide a better alternative, otherwise what is the point of the complain? If he don't have a better alternative, so he should rename to "JSON sucks" or something like that.

jfoster2• 12 years ago

You're right, but that's so easy to do unless the problem matters and is something that people are not easily able to realize on their own.

In this case, I'd say JSON not being the fastest it could possibly be doesn't matter too much. As a result, pointing that out isn't particularly valuable, but could be a bit more valuable if there was an easy-to-adopt solution also being proposed.

EC12• 12 years ago

If you're going to post against such a widely used format, you had better have a very solid reason, and an even more solid alternative.

rdtsc6• 12 years ago

I think he means not having something to replace in general doesn't mean not having something to replace it with in particular instances. Maybe there isn't a good general all-singing-and-dancing format.

There are language specific serializations if you never leave that language environment. There are protobufs, ASN.1, messagepack, thrift, bert, avro.

I am personally am not rushing to replace it but I can see the point he is making.

MagicWishMonkey• 12 years ago

Using something other than JSON is a great idea if you don't want anyone to ever bother using your API.

strmpnk3• 12 years ago

Content negotiation is a great way to expose multiple serialization formats. No reason it has to be just one of the well known formats. The main tricky part is equivalence of more complex structures in JSON, for example msgpack supports more than just string keys in maps.

hapless2• 12 years ago

So now I get to support more than one format, with corresponding increases in complexity...

..But only the JSON format will have any users, and I still have to deal with anything I find objectionable about JSON.

Tobani2• 12 years ago

The adage "The simplest things that could possibly work" comes to mind. For many things JSON is good enough.

Going with JSON first makes sense, if you need to create an alternate more efficient protocol in the future you'll at least have a few more data points to use when selecting something new like msgpack/protobuffers/thrift or whatever.

jackmaney• 12 years ago

Yes, because XML is so much more compact.

Oh, wait....

efuquen2• 12 years ago

I don't see xml mentioned once, I'm sure he would say it's worse. He'd clearly probably advocated some sort of binary format, since he mentioned msgpack. Did you even read the comment?

maaaats• 12 years ago

At least I can comment my xml config files. I hate that all my JS config is in JSON with no explanation in the file for why certain parameters were set.

rdtsc6• 12 years ago

Show me in that post where the word XML was used.

Oh, wait, you haven't read the post.

peteretep• 12 years ago

    > *  It has to be valid UTF-8, meaning it's incredibly
    > slow to validate.

If make basic mistakes about what the format allows, the rest of your comment is liable to be junk.

HugoDias2• 12 years ago

Ok, I understand your argument, now tell me a better format to create an API, Thanks ...

icebraining4• 12 years ago

Turtle!

robgering• 12 years ago

The author suggests using MessagePack, which I hadn't seen before but looks really cool.

http://msgpack.org/

tmikaeld• 12 years ago

It sure looks impressive, thanks for linking

jamieomatthews• 12 years ago

This does look pretty cool. I think the author is forgetting about so many reasons why JSON is so widely used, however.

One huge benefit of JSON over msgpack or something similar is READABILITY.

jwn• 12 years ago

So MessagePack wants me to ditch one of the most portable widely used data transfer formats in the world to save 33% on message size? No thanks. I'm willing to bet that using Content-Encoding: gzip on the JSON would negate most of the size difference anyway.

denzquix3• 12 years ago

But that also uses double precision floating point number representation and UTF-8 for strings, so the only benefit (according to the arguments given in the OP) seems to be text vs. binary-based.

craigching4• 12 years ago

But that also uses double precision floating point number representation

I don't think JSON requires the number representation to be double-precision, that can be up to the parser to decide how to represent a number. I think most people misrepresent this because the number implementation in JavaScript is double-precision, but that's not necessarily true for JSON. Am I wrong about that? Does JSON require the number format to be double-precision?

arethuza• 12 years ago

So don't use JSON when you need super high performance and be careful about parsing JSON numbers?

It may not be perfect, but compared to the complexity of the XML world and the opaqueness of binary formats JSON is a very pleasant compromise.

Shame about it not having comments though.... :-)

jfoster2• 12 years ago

This essentially amounts to "JSON is slower and more CPU intensive than it absolutely needs to be."

I can live with that. Any solution that people come up with, there will be folks who can point out some sort of flaw in it. Sometimes the flaws are worth paying some attention to, when fixing them might lead to some tangible benefits. It's difficult to imagine a slightly less CPU intensive replacement for JSON bringing any tangible benefits with it, though.

strmpnk3• 12 years ago

The note on numbers can be specifically problematic since the format does not specifically restrict numbers to be represented by doubles but many clients will assume so, creating some hard to find bugs (it's bitten me many times). The others seem more minor compared to its lack of things like maps that support non-string keys.

Turing_Machine• 12 years ago

Hmm.... isn't JSON required to be valid Javascript?

Javascript assumes all numbers are double-precision FP, yes?

icebraining4• 12 years ago

The lack of hypertext is a valid complaint, in my opinion, for a language designed in the web age. Sure, you can build a format on top of JSON that treats certain values as URIs, but that kind of thing should be built-in.

craigching4• 12 years ago

Just curious, but how do you envision hyperlinks working in JSON? And what's a use case for this? Generally I imagine putting "next" sorts of links in a JSON document is the way I resolve this, but I could imagine wanting to put a link directly on the text that would take you to more detail about the text. Is that the use-case? I guess I haven't needed that or I haven't known I've needed it because it isn't there :p

Tobani2• 12 years ago

I don't quite follow the whole hypertext problem. I mean URIs can be encoded as strings, there's no problem there.

HTML doesn't handle URIs any differently. They're just string attributes on elements(generally).

Is the problem that there aren't clients that can generally consume the JSON and find the hyperlinks? Is the idea that the JSON should be able to be rendered as a site kind of like the whole xml/xslt idea?

I guess I just don't get what the problem is here? What would built-in uri's look like?

Thaxll• 12 years ago

Servers aren't CPU bound anymore, there is no need to worry about that...

lpgauth• 12 years ago

Well, it depends on your scale.

sigmaml• 12 years ago

It depends a lot on the service that you provide. I provide compute services. Most of my servers are CPU-bound most of the time.

jerf2• 12 years ago

Erlangers are particularly grumpy about JSON because unlike many other languages, Erlang lacks a decent default serialization and deserialization of JSON. This is primarily due to the fact that a string is a list of integers, thus making it hard to distinguish between the two, and somewhat less importantly, but still problematic, the klunkiness of the dictionaries. (This message appears to be in the context of the soon-to-come native addition of dictionaries, but as I imply, unfortunately that is not the biggest problem with JSON deserialization, the nature of strings in Erlang is. Binaries are not a precise conceptual match to a JSON string either, unfortunately; in my environment I actually use binaries that way because I can guarantee the differences won't affect me since I control both ends of the protocol, but that is not safe in general.)

This is not to say that the arguments are wrong, but I think you may not understand why Erlangers feel the way they do until you realize that Erlangers also don't generally get to experience the advantages of JSON, either; when you get none of the advantages and only the disadvantages, anything looks bad, no matter how good it may be in the abstract.

This is a particularly rich irony in light of the fact that Erlang's native data model is conceptually the closest language I know to natively using JSON as the only data type. Erlang has no user-defined types, and everything is an Erlang term... imagine if JS worked by only allowing JSON, so, no objects, no prototypes, etc. The entire language basically works by sending "ErlON" around as the messages, including between nodes. It's just that there's no good mapping whatsoever between Erlang's "ErlON" and JSON.

rubyrescue2• 12 years ago

Erlang has amazing default serialization and deserialization. term_to_binary(T) and binary_to_term(B)

jerf2• 12 years ago

(I edited the last paragraph in before seeing your post here.)

angersock• 12 years ago

Right, but what about JSON?

Parsing the binary blobs for interop is annoying, see for exmaple the Wings file format, made almost entirely of Erlang serialized objects:

http://en.wikibooks.org/wiki/Wings_3D/User_Manual/Wings_File...

kungfooguru• 12 years ago

https://www.youtube.com/watch?v=wKjxFJfcrcA

deividy• 12 years ago

Oh, now that you mentioned it, I'll use XML, It's very cool and compatible with new technologies, like a new thing of Microsoft called C#.

nubs• 12 years ago

Its numbers representation is double-precision floating-point, meaning it's incredibly imprecise and limited.

Except that JSON actually just specifies the number type as an arbitrary precision decimal number.

Many implementations use floating point numbers when decoding JSON, but that is not inherent in JSON.

My biggest complaint about numbers in json is that often times floating point numbers get turned into integers when encoding/decoding using some implementations. (e.g. (float)2 gets encoded as just 2 and when decoding, it is an integer rather than a floating point.

vesinisa• 12 years ago

Well, regardless of the specification, most JSON deserializers approximate numeric values as floats, at least in the default configuration. Enough so, that the Bitcoin exchange JSON-APIs I know of all put numbers in strings. So `{"balance": "12.65334221", "currency": "BTC"}` instead of `{"balance": 12.65334221, "currency": "BTC"}`. It would easily be pretty catastrophic if there are any rounding errors in your currency calculations, and if the bug arises already in your JSON deserializer it might be hard to detect even if your program internally employs BigNums (arbitrary precision) or such.

jcizzle2• 12 years ago

JSON and Erlang just don't get along in the way you'd like them to, unfortunately. Some of that has to do with how Erlang handles (or doesn't handle) strings. Some of that was that there was not a data structure that closely mirrored JSON's structure in Erlang up until Maps were introduced.

The problem isn't really JSON - JSON is an exceptional format for what it is supposed to do - the problem is that Erlang was created for a specific purpose and that purpose wasn't to vend out strings/JSON over HTTP.

rdtsc6• 12 years ago

Erlang handles (or doesn't handle) strings.

JSON string are translated to binaries. There is no reason to keep repeating oh inters=strings. Maps will help.

The problem isn't really JSON

I think the author disagrees. He talks about the problems of "JSON". There are a few he notes:

* No binaries.

* No way to have hyperlinks

* Limited floating point number implementations

jcizzle2• 12 years ago

You have to translate your strings to binaries before you try to encode to JSON. A JSON encoder has no way of determining whether it should be encoding an array or a string without this. Instead of one string type, you have "strings" and <<"strings">>, and due to the lack of typing, it is actually pretty tough to keep track of which one you have. It also is an exercise in finger acrobatics to <<>> all your strings.

JSON doesn't solve every problem. It does solve the problem, quite elegantly, of creating a 'endianless' interchange format that is easily consumed, constructed, and debugged.

prottmann2• 12 years ago

That reasons exists since a decade with emails. And did we change the system ? No, because it is simple and it work.

hapless2• 12 years ago

Only an Erlang mailing list would prompt someone to complain that a document format requires valid UTF-8.

This is a glimpse of the last place on earth where relying on US ASCII is considered a positive good.

jnevill• 12 years ago

I was thinking to myself while reading the link "Why on earth would someone think UTF8 is a problem?" and then I saw it was a erlang thread. To complain about it being slow to validate while talking in an erlang thread... christ. It must be frustrating trying to deal with json, which is full of strings, in a language like erlang.

rdtsc6• 12 years ago

JSON decoders decode string to binaries.

https://github.com/talentdeficit/jsx

Erlang has been receiving and sending data. It just like to deal well defined binary messages and it likes to encode/decode them into its own representations (records, terms) at the boundary where they come in and leave the system.

ctz• 12 years ago

The thing is that JSON doesn't actually require or otherwise specify UTF-8. A JSON encoding is defined as a sequence of unicode codepoints: it doesn't actually tell you how to send those over the wire. You have to do that yourself, and specify the encoding out of band somehow.

If JSON did actually specify UTF-8 and provided a sensible escape mechanism for the whole of unicode, this would be an improvement, because it would turn JSON into an actual standard data-exchange format, rather than a textual encoding of its infoset.

derefr• 12 years ago

They're not talking about strings in JSON; they're saying that a JSON-serialized blob must itself be valid UTF-8. So, to decode JSON, you first have to parse the whole thing to convert it to codepoints (and possibly bail at this step)—and then (lex-and-)parse those codepoints again to turn them into a data structure, making sure to pass through things that turn out to be strings as the same sequence of codepoints you already parsed out rather than passing them to a UTF-8 decoder there and then, like you would in any other format.

Erlang's internal serialization format, meanwhile—and all the language's native pattern-matching constructs—are built on what are basically "generators" that consume a (possibly-infinite) binary stream, and lex tokens directly out of it. JSON doesn't really work with this approach; what you end up doing is having one generator that consumes the binary stream and emits codepoints, and then another generator that consumes codepoints and emits tokens. This introduces a lot of intermediate allocations and message-passing—whereas, with most Erlang protocol handlers, your TCP handler passes you a slice of a VM-managed shared binary, and then you just pass around and re-slice that slice.

antirez• 12 years ago

Prefixed length formats that don't make assumptions on encoding are a very good pick. The most important thing is that once you read the length, you can just bulk-read the payload without actually parsing it, that's up to the higher layer (if parsing is needed at all).

A general purpose serialization format that requires per-char processing is a terrible pick.

rdtsc6• 12 years ago

Agreed. I had to implement base64 encoding to pass data through JSON, yeah it works but seems terribly wasteful.

NathanKP• 12 years ago

The first requirement that I look for in a data interchange format is that it should be readable, because I don't want to look at binary data responses. The second thing I look at is support for encoding and decoding it.

The only three formats I've found so far that are readable and well supported are XML, JSON, and YAML. XML is too hefty and wasteful. YAML has had a bad history of insecure encoders and decoders but overall is my favorite data format. However, it still has the downside of needing a special decoder since browsers don't support it, and it requires specific indentation for its hierarchical data format which is wasteful in its own way.

That just leaves JSON in my opinion. It's easily understood and read, and native browser and Node.js encoding and decoding is more than fast enough.

famousactress• 12 years ago

...I don't want to look at binary data responses

I've always found this pretty semantic. Obviously tool support for reading ascii or UTF-8 encoded text is very strong, but it's all binary, so it's a question of tools. I do find JSON the most palatable of the text-encoded formats, but I'd adore it if a solid binary replacement gained favor. Unfortunately I think a lot of the protocols that have had pretty good energy behind them (pbuffers, thrift, hessian, etc.) end up either going the compiled-stubs route ,or bundle RPC, or both.

I'd really like to see a binary protocol with solid type support (please God, let it define a built-in datetime) that can be written and read dynamically (without a header file/stub).

kasbah2• 12 years ago

What about S-Expressions? My electronics CAD program recently switched to using it for footprint and layout files and I am very happy about it. It looks like this in text: https://github.com/KiCad/Footprint_Symbols.pretty/blob/maste...

peq• 12 years ago

Are S-Expressions a standardized data format? It looks very similar to Json, but harder to read.

patrickmay• 12 years ago

s-expressions (http://c2.com/cgi/wiki?XmlIsaPoorCopyOfEssExpressions) are another option. Even people who prefer JSON recognize the advantages: http://irreal.org/blog/?p=713

barrkel• 12 years ago

Sexprs are (usually) node-labeled trees (xml is too), while json describes edge-labeled trees. Object-oriented data structures are edge-labeled graphs. Json is a slightly better fit for the most common implementation languages.

The page you linked to doesn't support your argument, BTW. It tries to assert that sexprs support hashes - by extending the syntax with a reader macro!

This line shows the author's confusion: "S-expressions are more powerful (because of the duality of code and data)". The guy is confusing a format that strictly should not have behaviour beyond constructing a data structure, with Lisp more generally. If you allow the data structure to contain code that further interprets the data structure, the complexity of sanitizing input greatly increases.

rdtsc6• 12 years ago

I don't want to look at binary data responses.

I keep seeing this but having looked at hundred megabyte large JSON objects or XML files, some might as well be binary to my eyes.

alecthomas• 12 years ago

Agreed. And further to this, unless the JSON is indented for humans (not common) you still need a tool to format it. So if you're using a tool anyway, you could just as easily have used a binary format.

craigching4• 12 years ago

The only three formats I've found so far that are readable and well supported

It probably doesn't fit your "well supported" criterion I suppose, but how about Clojure's EDN? I've been meaning to try it out more myself.

mwcampbell• 12 years ago

I always use "python -m json.tool" to pretty-print JSON data before looking at it. I could just as easily do the same with a MessagePack or BERT pretty-printer.

draegtun2• 12 years ago

Let me also add Rebol into the mix. To quote Carl Sassenrath...

Every time I run across JSON examples, I see REBOL without the elegance. The two languages are related of course. REBOL strongly influenced the design of JSON.

ref: On JSON and REBOL - http://www.rebol.com/cgi-bin/blog.r?view=0522 (HN - https://news.ycombinator.com/item?id=5654895)

denzquix3• 12 years ago

Since the author seems to have a problem with text-based formats in general, here's a counterpoint by Mike Pall on the LuaJIT mailing list [1]:

"On a tangent: IMHO most binary serialization formats like BSON, MessagePack or Protocol Buffers are misdesigns. They add lots of complications to save a couple of bytes. But that does not translate into a substantial improvement of the parsing speed compared to a heavily tuned parser for a much simpler text-based format."

[1] http://www.freelists.org/post/luajit/Adding-assembler-code-t...

Zikes• 12 years ago

If msgpack is able to match JSON for serialization/deserialization, couldn't you make an API in JSON and offer msgpack as an alternative format?

rhizome31• 12 years ago

In a project I'm working on, we do that with Protocol Buffers. Our main client uses protobuf but we still provide JSON for other clients. We also use the JSON API for debugging.

efuquen2• 12 years ago

It's text-based, meaning it's incredibly slow to parse.

This is precisely the reason it is used so heavily. Easily readable format to reason about. And incredibly slow is a pretty relative term, when you're waiting on database calls or doing other complex logic that is orders of magnitude slower, who cares how "slow" json parsing is.

It has to be valid UTF-8, meaning it's incredibly slow to validate.

I think being able to embed all sorts of different characters and languages is, again a plus. See argument above about performance.

Its numbers representation is double-precision floating-point, meaning it's incredibly imprecise and limited.

This argument I don't get. I'm pretty sure I might be missing something, but in my experience you can just put a plain old integer and any parser in any language will extract as an exact integer. Nobody is converting a "1" in json into a double/float. Maybe somebody can elaborate and what the author might have actually meant?

So, really, the argument all boils down to, it's slow and wasteful. Well, while that's true I think it's pretty much been established time and time again that Moore's law has made it possible to value programmer time over CPU overhead, to a reasonable extent (i.e. if the overhead you're adding overtakes Moore's law and makes infrastructure particularly expensive). If you have a format that is human readable, easy to understand, and simple, that helps tremendously in software development and it would take order of magnitude performance hits to really make it bad tradeoff (and even the, if you weren't getting a lot of traffic, who would care?), not just 2x or 3x.

We live in a web based world, and that world is fundamentally based on a text based protocol (http) and text based messaging formats. There are plenty of valid and good reasons why it happened the way it did.

*Final Note: I would like to see some empirical evidence to compare the wasted CPU cycles and energy that JSON uses compared to if all messages were sent with msgpack or something like it instead. While my own inclination is that number would be dwarfed by the overall energy used in computation I would prefer to see evidence rather then conjecture if you're going to make a point like that.

craigching4• 12 years ago

> Its numbers representation is double-precision floating-point, meaning it's incredibly imprecise and limited.
Maybe somebody can elaborate and what the author might have actually meant?

I think JSON numbers are not necessarily double-precision, at least I don't think I've ever seen them specifically described this way. In JavaScript, however, numbers are double-precision, there are no real integers in JavaScript. So maybe the author misrepresented JSON numbers because JavaScript numbers are double-precision. But that isn't necessarily the case AFAIK.

prottmann2• 12 years ago

Here the reasons why i use JSON:

* It's text-based, meaning it's readable and easy to parse.

* It has to be valid UTF-8, no hassle with other formats.

* Its numbers representation is double-precision floating-point, i can choose what i want.

Who cares? Use what you want and what solves your problem at the best way.

No JSON/MAPS interoperability in 17.0?

Related Stories