The Rise and Rise of JSON (2017)

https://twobithistory.org/2017/09/21/the-rise-and-rise-of-json.html

by mariuz • 6 years ago

89 163 6 years ago

twobithistory.org

jsnell• 6 years ago

(2017)

Previous discussion: https://news.ycombinator.com/item?id=17832936

Zamicol• 6 years ago

JSON is one of the best things to ever happen to software development.

abraxas4• 6 years ago

It really isn't. I'm almost tempted to say it's the opposite though that would be overstating the case.

It's a format that still bears excessive decoration (what's the purpose of quotes around field names? what are all those commas for?) yet it's limited in the types of data structures that it's able to express (natively). I'm not particularly fond of Clojure specifically but a format like EDN would have been superior in just about every way.

hackcasual2• 6 years ago

I'd hardly call a few bytes per key and field excessive. Especially compared to something like XML.

The datastructure complexity being limited is also a pretty significant key to its success. More complex datatypes means greater chances for JSON handling libraries to lack compatibility.

The only substantial shortcomings of JSON I see are shortcomings associated with any textual serialization format. Optimizing for human readability in a use case that's 99.99% of the time not read by a human.

kanobo3• 6 years ago

Quotes allows for any characters and limited types is probably one of the main reasons it's widely used and implemented. There's no doubt vastly superior formats tailored to specific languages and purposes, but I think it's hard to argue that JSON was not a huge net positive to the software industry as a whole.

baddox2• 6 years ago

Listing some specific perceived flaws with JSON isn’t counter to the claim that it’s one of the greatest things to happen in software development.

fomine3• 6 years ago

Lack of integer type is really suck but that's inherited failure from JavaScript.

redleggedfrog• 6 years ago

It'd be nice to know what the graph would like like now, three plus years later. I imagine it kept along the same lines. JSON is everywhere.

I suspect that a lot of the popularity is piggy-backing on the popularity of JavaScript.

But what the heck is .CSV doing on an incline?!

crooked-v2• 6 years ago

CSV generation and parsing is inconsistent based on tooling (see inevitable space/comma/quote problems, for example), while JSON has a strict enough spec that you can unambiguously use it in the same way for CSV-like data as long as nobody needs to literally edit it directly in Excel.

ciupicri• 6 years ago

CSV has a specification too - RFC 4180, "Common Format and MIME Type for Comma-Separated Values (CSV) Files" [1]

[1] https://tools.ietf.org/html/rfc4180#section-2

goatinaboat• 6 years ago

CSV is so unnecessary - ASCII has built in delimiters for fields outside of the printable characters.

teddyh6• 6 years ago

CSV is for human-readable data. If you don’t need to be human-readable, you might as well go with ASN.1 or something.

jjoonathan5• 6 years ago

Text tools have intermittent support for those which would destroy the one big advantage of CSV.

Once you're using a binary format, you might as well use something like parquet or hdf5 (or ASN.1 I suppose) that give you other goodies in addition to airtight data/organization separation.

jjoonathan5• 6 years ago

CSV forces struct-of-arrays and forces each row to have the same structure. Those are sometimes useful constraints. Also, you can obtain it from (legacy app) and open it in excel.

sradman• 6 years ago

what the heck is .CSV doing on an incline?!

It is still the most common interchange format for tabular data. Tabular data is too verbose/unreadable when represented in standard JSON. Databases and Data Science are also on the rise; .csv continues to ride this wave.

hombre_fatal3• 6 years ago

Also, "Why use something else when we can use data.join(',') -- that should be enough!" must also be a contributing factor to why CSV will never die.

mr_toad• 6 years ago

You can use column-oriented JSON (with an array for each column of data instead of each row), which is roughly the same size as a CSV file.

The downside is that you can’t process a row at a time.

kanobo3• 6 years ago

Great article. It's strange for me to be old enough to have experienced JSON's entire history as a website maker - I didn't learn anything new in the article I didn't know before and never gave it much thought - but to read it all compiled in one place makes me want to appreciate that we are experiencing tomorrow's history right now.

amelius5• 6 years ago

My only problem with json files is that you can't add comments to them in a persistent way (especially if changes in the data are written by a program).

Unless of course you'd put the comments in the data itself, but that's kind of ugly.

k__5• 6 years ago

Also: streaming

amelius5• 6 years ago

Can't you implement streaming by just putting everything in a giant JSON list?

Or do you want to send different parts of e.g. a map in separate streams?

switch007• 6 years ago

There’s jsonlines but it is not widely supported. Eg PowerBI has to be told to treat it as CSV then convert each row in to JSON rather than a native jsonl type.

thwarted• 6 years ago

No worse than putting data in comments. The CDDB format was notorious for putting track length and some other details in comments, and a fully supported parser then had to be smart enough to extract data from them

crooked-v2• 6 years ago

You might like JSON5 (https://json5.org), which allows comments and has direct support in various tooling.

amelius5• 6 years ago

Yes, thanks, but are the comments stripped out when you read the JSON? Or are they still accessible, so they can be written back when a program modifies the JSON slightly?

ehsankia• 6 years ago

There are many json alts, but at the end of the day unless it's supported everywhere, it kinda defeats the point, because what makes JSON great is that it's supported almost everywhere.

benhoyt• 6 years ago

It's true, but I've used patterns like this to good effect:

    {
        "_COMMENT": "default; override with --s3-bucket",
        "s3_bucket": "foo-bar"
    }

tandav• 6 years ago

my personal pain is that you can't add trailing comma (PAIN!!!)

aaron-santos3• 6 years ago

Personal opinion: putting comments in the data is a smaller sin than putting data in comments. More information @ https://news.ycombinator.com/item?id=3912149

amelius5• 6 years ago

Ok, but would you prefer:

    [1, 2, 3, 3.14159, "comment: Pi!!", 4, 5, 6, 7, 8]

over:

    [1, 2, 3, 3.14159 /* Pi!! */, 4, 5, 6, 7, 8]

throw0101a5• 6 years ago

My only problem with json files is that you can't add comments to them in a persistent way

JSON was first designed as a data exchange format between computers (that was lighter weight than XML (which was lighter weight that SGML)), so that fact that it's being used in persistent fashions is getting away from its primary purpose.

rumanator2• 6 years ago

I would argue that any human-readable format that relied on defining blocks by opening/closing symbols is a format that's not suited for config files that are expected to be manually edited by humans with a text editor.

teddyh6• 6 years ago

no new version of the JSON specification is ever expected to be written.

T̵h̵e̵ ̵l̵a̵t̵e̵s̵t̵ ̵v̵e̵r̵s̵i̵o̵n̵ ̵o̵f̵ ̵J̵S̵O̵N̵ ̵w̵a̵s̵ ̵a̵p̵p̵a̵r̵e̵n̵t̵l̵y̵ ̵r̵e̵l̵e̵a̵s̵e̵d̵ ̵l̵e̵s̵s̵ ̵t̵h̵a̵n̵ ̵a̵ ̵y̵e̵a̵r̵ ̵a̵g̵o̵.̵ [EDIT: I was mistaken: the latest standard was RFC 8259. However, this was still published on 2017-12-13, after the article was written.] There have been three different RFCs alone, all defining JSON. (And let’s not even get into the thing called JSON5.)

seagreen4• 6 years ago

Which specification got an update last year? Google's not turning anything up.

kozak• 6 years ago

I've tried to find out what was the change less than a year ago, and I couldn't find this information. So, what was the change?

teddyh6• 6 years ago

I seem to have misread my reference: the change a year ago was to Javascript, to make it conform to JSON, not the other way around, which was how I originally interpreted it. The latest version of JSON itself seems instead to be the latest RFC from 2017-12-13. Note: this was still after the article was written.

throw0101a5• 6 years ago

There were six changes outlined in RFC 8259, three of which were moving from ECMA-262 to make ECMA-404 a normative reference, one was to explicitly state UTF-8 should be used in a particular section, and one was 'to increase the precision of the description of the security risk that follows from using the ECMAScript "eval()" function':

* https://tools.ietf.org/html/rfc8259#appendix-A

Not very Earth shattering revisions IMHO.

k__5• 6 years ago

I remember 2010, when I build my first API. People laughed at me for using JSON.

I had to add XML and CSV, because "nobody would integrate with a JSON API"

chc• 6 years ago

I think whatever company you were working at was a few years behind the curve, because JSON was already pretty much the standard for SPAs by 2010. For example, that's the year AngularJS came out, with JSON/JSONP as the only built-in serialization format for communicating with backend APIs.

k__5• 6 years ago

Pretty much, yes.

In 2011 we moved from an PHP app to an SPA where JSON came in handy.

falcolas3• 6 years ago

JSON won, IMO, because it's human readable and writable. It came at a time when its main competitor in this space - XML - had gotten too complex.

However, JSON is hitting the same limitations and problems that XML faced, and is following in their shoes (namespaces, schemas, x/jpath, implementation drift across libraries, etc).

jjoonathan5• 6 years ago

Yeah, but they were reinvented in JSON, with the key difference that you didn't have to pay for them if you didn't need them. JSON unbundled XML.

Spearchucker2• 6 years ago

The same was true for XML.

kerkeslager6• 6 years ago

Ehhhh, is it hitting those limitations? I've yet to run into a problem in which would be solved by adding Enterprise™ silliness.

And is it following in XML's shoes? I've not had to work on or integrate with any systems that did this Enterprise™ silliness either.

Now granted, I'm fairly separated from the world of IBM and SAP type companies, and I'm sure there's all sorts of unholy abuse of JSON going on in a dark server room somewhere, but the answer in most cases is just to choose technology stacks that don't do that to you. These problems are generally self-imposed rather than imposed by the technology.

falcolas3• 6 years ago

Ehhhh, is it hitting those limitations?

Yes. Some folks, even at my relatively small company, are using JSON and also pushing for extensive schema use.

JSON - just like XML - allows you to use its vanilla form. But the additional tools are available if/when you need them, because the need exists.

kstrauser• 6 years ago

Ehhhh, is it hitting those limitations? I've yet to run into a problem in which would be solved by adding Enterprise™ silliness.

I think that's why JSON won: it was first and foremost and engineer-driven standard as opposed to one that incubated inside a bunch of enterprises before being dropped on an unsuspecting world.

krapp2• 6 years ago

JSON won because the web and javascript, which supports JSON, won. It would have won regardless of how ugly or awkward it was for that reason alone.

iakh• 6 years ago

Except that the x in ajax stands for xml

enricozb2• 6 years ago

I wish JSON had 1) trailing commas 2) comments (I'm not sure if this is a good idea or not but every once in a while I impulsively write comments)

kanobo3• 6 years ago

It's funny I have the opposite feeling. In my case I write a formatter/validator in each project for my json that fixes issues like trailing commas and is also where I annotate the data structure, but I can imagine for those who write JSON by hand those two limitations would be a headache.

tgb• 6 years ago

I just wish it supported all floats (nans and infinities too).

nitrogen2• 6 years ago

An integer datatype would also be very useful. 53 bits isn't always enough.

abraxas4• 6 years ago

3) set types, 4) optional schemas 5) multiline strings 6) consistent support for numeric types 7) streaming data

The litany is not short and I'm sure I've overlooked a couple.

selectnull• 6 years ago

8) date types

speedgoose2• 6 years ago

You can try json schema. Streaming data is also easy with json lines.

But if you add too many features, it may end up like XML.

austincheney• 6 years ago

JSON is popular because it’s tiny. If you want to trade tiny for a litany of features and data types then just use XML.

aloisdg• 6 years ago

You should try XML. You may like it.

emil0r• 6 years ago

So basically EDN.

throw0101a5• 6 years ago

4) optional schemas

Perhaps CDDL:

* https://tools.ietf.org/html/rfc8610

user59944614• 6 years ago

What do you mean by schema? JSON supports schema with libraries included in most popular languages nowadays. https://json-schema.org/

For numeric types, if you handle integer larger than 2 billions or floating numbers where the precision is critical (that's where json fails), these can and should be represented as string. I don't think there is any common format that guarantees large numbers and/or arbitrary precision to be encoded and decoded properly across languages and platforms.

seagreen4• 6 years ago

Comments are a great idea... for a configuration format!

They're really bad for a data interchange format though, because people inevitably start putting important data in them and you end up with two different ways to write strings, one of which isn't supported by every parsing library.

Thus each discussion of JSON ends up being two groups talking past each other, the people using it as a configuration format who lament the lack of comments, and the people using it as a data interchange format who celebrate it.

The solution: since it's too late to add comments now, don't use JSON as a configuration format.

Tokkemon2• 6 years ago

So Yaml?

wayneftw• 6 years ago

JSONC is a thing and has both of those features. VS Code used it all over the place.

tabtab• 6 years ago

Re: "[Comments are a] bad for a data interchange format though, because people inevitably start putting important data in them"

Any feature or tool can be abused and misused. Comments are useful for configuration files, period.

Tokkemon2• 6 years ago

Yes, definitely. Every time I work with PHP Arrays, I'm reminded both these things don't exist in JSON and it makes me sad.

dhbradshaw• 6 years ago

The other day I saw an auto-generated json file with comments and trailing commas and got really excited. A little digging showed that it was using JSON5. Pretty sweet!

https://json5.org/

divbzero2• 6 years ago

Cool! JSON5 looks intriguing.

The core project implements a JavaScript module [1] but it looks like there are also experimental JSON5 libraries for Go [2], Python [3], Ruby [4], and Rust [5].

[1] https://www.npmjs.com/package/json5

[2] https://github.com/yosuke-furukawa/json5

[3] https://pypi.org/project/json5/

[4] https://github.com/bartoszkopinski/json5

[5] https://docs.rs/json5/

I noticed that JSON5 doesn’t include a Date type. There have been discussions about this in both the JavaScript [6] and spec [7] repos but with no resolution yet.

[6] https://github.com/json5/json5/issues/3

[7] https://github.com/json5/json5-spec/issues/4

smabie• 6 years ago

HOCON solves pretty much all the problems JSON has, including the ones you mentioned.

throw0101a5• 6 years ago

2) comments

Per Douglas Crockford, the creator of JSON, comments were an anti-feature:

I removed comments from JSON because I saw people were using them to hold parsing directives, a practice which would have destroyed interoperability. I know that the lack of comments makes some people sad, but it shouldn't.
Suppose you are using JSON to keep configuration files, which you would like to annotate. Go ahead and insert all the comments you like. Then pipe it through JSMin before handing it to your JSON parser.

* https://web.archive.org/web/20120507093915/https://plus.goog...

Discussion on the post (2012):

* https://news.ycombinator.com/item?id=3912149

Remember: JSON was designed primarily as a data exchange format between computers.

MichaelMoser123• 6 years ago

Processing instructions can always be added as a special name value entry, that wouldn't be a good reason to get rid of name value pairs

enricozb2• 6 years ago

This is what I figured would be the reason to remove comments, because they would hold some other non-human information. It's unfortunate that people do this. The main reason for comments for me would be for config files, which maybe shouldn't be in JSON anyways.

user59944614• 6 years ago

There you go json for humans https://hjson.github.io/

This allows trailing commas and comments. This can also reads strict json of course. Libraries are available for all popular languages.

raverbashing• 6 years ago

Amen

XML is a mess and a chore to work with (at least in any language that isn't Java I guess, but even then).

Yes, it has some rough edges. Yes it could be better.

But overall it's good. Not too complicated and not too hard. Works fine for most stuff.

enriquto2• 6 years ago

In 10 years we will see JSON with the same mix of mockery and regret as we see XML today.

scrollaway• 6 years ago

I remember reading that opinion 10 years ago.

seagreen4• 6 years ago

Just curious-- have you actually read the specifications for both? The difference in length and complexity is like this comment vs. War and Peace.

atarian• 6 years ago

If another standard were to dethrone JSON, my guess would be YAML.

timw4mail• 6 years ago

Considering that the YAML spec rivals XML in complexity, I doubt it. Between that complexity, incompatible parsers, intentional loose types...what do you really gain?

jniedrauer• 6 years ago

YAML will never be a sane format for machines to talk to each other. There are just too many ways to interpret it. It's not even really a good human-readable format.

Maybe something like TOML? It's very human-readable and simple. But it's not a very good serialization language.

wvenable4• 6 years ago

I remember using XML-RPC back in the day to communicate between a desktop client and a web service. It was fantastic. And then designer went off with a committee somewhere and produced SOAP. The regret with XML is that it continued to evolve into a monstrosity.

Thankfully JSON came along and got back to the simplicity of early XML and XML-RPC and stayed there.

user59944614• 6 years ago

I remember killing some old XML-RPC applications.

Deserialization and remote code execution vulnerabilities all over the place. That was brutal.

Who thought this was a good idea to pass arbitrary function names and arguments for the remote servers to resolve and execute blindly? The regular vulnerabilities in the XML parsing libraries themselves was the nice cherry on top.

enriquto2• 6 years ago

I remember using XML-RPC back in the day to communicate between a desktop client and a web service. It was fantastic.

My favorite quote about XML-RPC (from the plan9 people):

Some part of me desperately wants to believe that XML-RPC is some kind of elaborate joke, like a cross between Discordianism and IP Over Avian Carriers

I have exactly the same feeling regarding the JSON "protocols" and whatnot.

zeveb• 6 years ago

It's a real shame that both XML and JSON eclipsed the real data format for the ages: S-expressions. I hope it's only temporary, though: there really is no good reason the prefer JSON other than everyone else using it, and if one gets into a novel enough domain then there isn't an everyone else doing anything to worry about yet.

S-expressions have the advantage of not containing the redundant object type (there's no need for an object, map or dictionary when one has alists or plists), and the even greater advantage of elegantly representing code.

Also they are easier to write parsers for.

teddyh6• 6 years ago

There was this: http://people.csail.mit.edu/rivest/Sexp.txt but it was never even accepted as an RFC, it seems.

blackrock3• 6 years ago

Very interesting. Did anyone else look at this, and think it was super complicated, at first glance? You’d have to be an expert in the data exchange format itself.

I can see why Lisp style syntax is bad for human comprehension. It’s because of the lack of an explicit delimiter, like a comma. I don’t think human eyes are very good at using a single space character, as a delimiter.

And with this format, you’d have to agree upon the data exchange structure. There would have to be a master key somewhere, maybe as the header. Albeit, this format is far more efficient than XML.

But, XML probably won out because it was explicit, and allowed multiple levels of nesting, but at the expense of overly wordy delimiter tags.

And JSON made it simpler, by forcing it to be a lightweight key-value pair.

It’s too bad something like this didn’t take root and become more popular. This might be a very useful advanced data exchange format for some scenarios. Although the headaches and problems associated with trying to understand, and work with it, might far outweigh its efficiency benefits.

blackrock3• 6 years ago

Please provide more details, insights, anecdotes, and examples of why you think S-expressions are superior.

olejorgenb• 6 years ago

A annoying problem with json is the lack of full floating point number support. Transferring NaN, +Inf/-Inf is a pain.

hajile2• 6 years ago

Crockford believes the One True Number should be decimal instead of floating point. He also believes that comments will be abused and turned into ad-hoc parser directives.

So everyone gets to suffer for his beliefs.

velox_io• 6 years ago

I'm quite a big fan of json.org it outlines the JSON specification in a handful of diagrams and a small amount of text, it's so simple and elegant.

reggieband• 6 years ago

I remember working with SOAP and WSDL. One advantage I kind of miss from that era was the static generators that would create your service classes and the strongly typed DTOs based on the WSDL. I understand some effort has been made to replicate this (e.g. json-schema) but it isn't nearly as widely used as WSDL seemed to be. To be honest, I only kind of miss it since XML was such a major pain to deal with (any one ever had to deal with XPath or XSLT? What a nightmare those were).

I recently used GraphQL on a project and it had some nice advantages. I love the idea of protocol buffers but have never had the chance to use them in anger. But if I'm honest, the boring option is JSON and it is what I would use for just about any API I had to expose nowadays.

66fm472tjy7• 6 years ago

One reason in favor of using JSON over XML for web services I don't see mentioned often is that many XML schema based deserializers will by default fail if they get an unexpected element whereas JSON deserializers will ignore it, making it easier to remain backwards-compatible. Example: If you have the following defined in your .xsd

  <complexType name="MyMethodResponse">
   <sequence>
    <element name="A" type="string" />
    <element name="B" type="string" />
   </sequence>
  </complexType>

and you later add a "C", existing clients will fail, so you will have to create a new method along side the old one to remain compatible.

willcipriano• 6 years ago

I've always wondered why the people who designed these things thought that I wanted a exception to be thrown in that case. It seems to be a "enterprise" feature, Java SQL libraries often blow up if they find a column that isn't predefined.

rahilb• 6 years ago

JSON was clearly built with XGH axioms in mind: ‘errors only appear when you notice them’.

elcritch• 6 years ago

Following proper XML spec always led to more problems than what they were meant to solve. Sometimes I use XML, but just the syntax with a more html5 style to it.

Similar to defaulting to error-ing on extra tags, I never got the point of making everything an element and adding extra redundant attributes. That's really just the result of XML that's automatically generated.

Like the `<element... type="string">` in your example of classic XML, why not just use `<a>3</a>` or `<a int="3"/>`. XML proper styles still suggest the classic long/autogenerated form. Writing XML in html5 style like `<my-a data-value="3" />` makes it much friendlier. Or as I tend to use it for various internal protocols to wrap multiple CSV data in a file:

    <some-data type"csv" columns="A, B, C">
      1.3, 3.4, 5.6
      1.3, 3.4, 5.6
    </some-data>

Technically That's really just html5 in a file I guess. :-) But html parsers also don't tend to complain about extra tags too.

thelazydogsback• 6 years ago

For small, heterogenous structs like messages and config it makes sense -- for repeated/homogenous collections (array of similar struct - a common case) both JSON and XML are wasteful -- a reflective/inline schema combined with a table would be more readable and space efficient. Also, meta-data needs to be either embedded or provided by special keys, so general meta-data tagging support would be great, where a non-meta-data-aware reader would only get the target data, but any JSON datum could be tagged with another JSON datum as meta-data. (The meta-data tags could be implement the inline-schema for the table as well.)

akubera• 6 years ago

I still think it's a shame that Amazon's Ion format never got widespread adoption.

https://amzn.github.io/ion-docs/

Superset of JSON with many extra features that people in the comments here desire from their data-language, such as support for comments, timestamps and s-expressions.

The_rationalist• 6 years ago

This is such a painful contingent historical accident that we have json as the standard instead of hson

doctoboggan• 6 years ago

This is a great bit of internet history, if for nothing else than this Douglas Crockford quote in response to a flamebait-y argument about XML being superior to JSON:

“The good thing about reinventing the wheel is that you can get a round one.”

atticusCr• 6 years ago

Although it is easier to see the benefits of using JSON as an interchange format because it is lightweight, I still believe that XML is more elegant and verbose than JSON. One of the complains for JSON has been the lack of schemas, although there are some ways around that in projects such as Apache Avro https://avro.apache.org/, registry schemas and all that Jazz

The Rise and Rise of JSON (2017)

Related Stories