Show HN: JSON5 — modern JSON

by aseemk • 14 years ago

64 84 14 years ago

aseemk15• 14 years ago

Author here.

This idea stemmed from long-running frustration every time I hand-wrote JSON (particularly common for Node/npm packages and test cases). After stewing on it for over a year, I bit the bullet today and made it a reality.

Feedback greatly welcome!

thwarted• 14 years ago

Why are you hand writing JSON? Pass a JavaScript object, which doesn't have these limitations, to the JSON serialerizer.

The support for evaling in the various engines doesn't matter because you should not be evaling JSON strings for security purposes.

The serialization format is strict for a reason: look at where less strict HTML parsing has gotten us.

aseemk15• 14 years ago

Writing, or at least editing, JSON by hand seems unavoidable if you develop on Node, as modules' "manifests" are JSON files.

I also love to make tests data-driven where possible. JSON is the natural format for data on Node, and again, test cases are usually written by hand (before code exists to generate or validate them).

I'm not sure I understand your point about eval'ing. This parser doesn't use eval(), and it works in every modern browser.

Finally, I'm not advocating for arbitrary looseness. These additions conform directly to ES5's additions.

stock_toaster• 14 years ago

* Objects and arrays can have trailing commas.*

If I had a penny for every time a trailing comma in a json config file bit me, I would have...many pennies!

I am indifferent about single vs double quotes. The backslash newline escape seems 'messy'.

Not a fan of keys not being quoted. That stikes me as wrongish for some reason that I can't put my finger on.

benatkin2• 14 years ago

Please don't call it JSON. Actually I don't care much, as your project will be shunned. The first rule of JSON is that JSON doesn't change.

gurkendoktor• 14 years ago

I absolutely hate hand-writing JSON, but that's why I use YAML. I think you should push for a properly documented subset of YAML, or a full YAML parser in JavaScript instead. Just a Ruby programmer's 2 Eurocent :)

aseemk15• 14 years ago

Thanks for the feedback! I never considered YAML precisely because there doesn't seem to be much support for it. I'll definitely look into it. =)

brentashley• 14 years ago

Douglas built the JSON spec

1) to be language agnostic. Following ES5's syntax directions is fine for ES5 focused development, possibly not for others.

2) to be final - i.e. there is no version number. If you want to extend or build something else, I am fairly certain Douglas would prefer that you call it something else. See http://inkdroid.org/journal/2012/04/30/lessons-of-json/ at 5:30 in the video.

Don't take these points as criticisms, I too would like to be able to use single quotes and trailing commas and everything else here. I think though that in order to work with JSON as a static spec, an alternative approach could be to have your project be a layer like HAML, SASS or CoffeeScript. Each of those tools compiles or transforms to conforming input for their target specs rather than replacing them.

aseemk15• 14 years ago

JSON5 is also meant to be language-agnostic. It derives its syntax from ES5 in the same way that JSON derived its syntax from ES3.

I'm aware that JSON is final — it's my hope that a new format that's easier to write (whether this or something else) picks up steam — but thanks for the feedback on the name! I consciously used a different file extension (.json5) to avoid conflicts; hopefully that's a good start.

I already use (and love) CoffeeScript, but my motivation for building this was to have tools use it natively, so that sibling files in a different format wouldn't have to be maintained alongside the needed JSON.

melloclello• 14 years ago

Horrible. There's a reason JSON is homoiconic with valid javascript. You should never have done this.

wvenable• 14 years ago

JSON is an amazing serialization format but hand-written JSON is completely wrong.

I think this fixes most of the problems of using it in hand-written situations. Just the lack of comments in JSON makes it inappropriate for configuration files and tests.

But this shouldn't be called JSON. JSON is strict and doesn't include comments because it's designed for interoperability between different systems. Just because it's been abused to be used for configuration files doesn't mean it needs fixing. One just needs to use the right tool for the job. Your project might be the right tool but then it's not JSON.

cheeaun• 14 years ago

JSON 3 here: https://github.com/bestiejs/json3

nfriedly• 14 years ago

I believe Facebook recently switched to JSON 3 for their JS SDK.

simanyay2• 14 years ago

Three things:

* Comments in JSON files are not a great idea because people might (will?) abuse them to add special instructions for their own serializers. As soon as this happens we'll have a format that is not compatible with other serializers.

* JSON today is a universal format. So, what are special characters? Should '1hello' be quoted? If no, then its not a subset of JavaScript anymore. If you're writing a new specification you have to define all those things. Same with multiline strings: space after backslash?

* Why? I know that hand-writing JSON can be annoying but is this really the biggest problem we have?

aseemk15• 14 years ago

Comments: yes, I read Douglas Crockford's post on the matter. I guess I'm not genuinely worried about that, though, and my hope is that shipping a JS parser that works on both the client and the server will help standardize implementations from the start.

Special characters: sorry I didn't write a formal spec =), but the intent is to be a pure subset of ES5 just like regular JSON is a pure subset of ES3. Unquoted object keys in ES5 can contain only letters/numbers/_/$, and only begin with letters/_/$. So yes, '1hello' would need to be quoted, just like regular ES5.

I'm not sure how to respond to your why other than what I've already written. I never claimed this was the biggest problem we have. =)

Too2• 14 years ago

JSON is not exactly tied to javascript. For example the number-type is not specified in much detail. You could imagine this to be a 64bit int but if you load it into javascript it will be converted to a double and you will loose precision if your number is big.

simanyay2• 14 years ago

Crock made a post about JSON comments?

Anyway, I wouldn't focus on JavaScript engines because JSON is now a universal format so for languages other than JavaScript you will have to define a formal specification.

zacharyvoase• 14 years ago

Can I ask why YAML wasn't good enough?

chc2• 14 years ago

My thoughts exactly. YAML is a superset of JSON that already has all this stuff and more.

aseemk15• 14 years ago

I actually had no idea that YAML is a superset of JSON. Thanks! I'll look into it.

erikpukinskis• 14 years ago

Partly because JSON has the lovely feature of being able to just eval() it into an object in browsers.

alcuadrado• 14 years ago

That's not the right way to parse JSON, it's really insecure

chc2• 14 years ago

That's not really a lovely feature in this day and age. If anything it's an unfortunate artifact, as the right way to deserialize JSON is JSON.parse().

silentbicycle• 14 years ago

For the same reason sexps weren't good enough, apparently.

Mizza• 14 years ago

Please, don't call it JSON. There is only one JSON, and you didn't make it. It's cool that you made this, and that you made it open source, just please call it something else.

enneff2• 14 years ago

JSON is wonderful because it is simple and unambiguous. It just works, and serialized data will always look the same.

This "JSON5" thing is an abomination. Use whatever quotes you like! More string escaping rules! Adding these things would make JSON worse, not better.

Here's another idea: why not write a tool that tidies up your hand-written JSON files? You could even re-use your "JSON5" parser to do it. That way you can be as lazy as you like when writing files, but not expect the rest of the world to learn your new dialect.

aseemk15• 14 years ago

Thanks for the feedback. It certainly would be great if there existed tools to let me write and update JSON with nicer syntax like this; any ideas how that might be possible?

My motivation for this was actually hoping that this gets adopted in some environments that use JSON. Node relies on package.json files, for example, which frequently need to be hand-edited. Maintaining a sibling file in a different format feels equally wrong to me as it seems JSON5 seems to you. =)

benatkin2• 14 years ago

That's npm, not node itself, that relies on package.json. When its creator chose it, he knew exactly what JSON is. He's maintained on HN and elsewhere that he made a good decision. So even if you stop calling this JSON, he's not buying it.

The package.json files, by and large, don't have a multiline string problem. For trailing and missing commas, I think you could make something that automatically fixes them, if that's the only problem, and run it before npm is run, perhaps using a shell alias.

tferris2• 14 years ago

Go and read why Crockford designed JSON like it is. E.g.: He omitted comments on purpose in order to avoid that people used them for additional directives.

Two or three "improvements" like quotes, comments don't legitimate a new standard. BTW, you could also leave out the colons.

enneff2• 14 years ago

It's not JSON, so why does it need to look like JSON? If you want a complicated human read-writable config format, it exists already: YAML.

As for your question: of course it's possible. Write a tool that translates between your preferred format and JSON. Set up your development environment to automatically convert between the two formats when editing JSON files.

exDM69• 14 years ago

> It certainly would be great if there existed tools to let me write and update JSON with nicer syntax like this

The syntax is not necessarily nicer, it has only a few changes in the lexical structure and the syntactic structure is pretty much the same. If it's a little nicer for a human writing it, it's a little hairier for a parser to parse it.

But how much you spend time writing JSON anyway? Most of the time JSON is read and written by tools. Human intervention is rare.

In addition to that, the native JSON parsers in the browsers are probably implemented in native code, and this parser of yours is written in JavaScript. So it wouldn't be a surprise if this was an order of magnitude slower than a native JSON parser.

mcherm• 14 years ago

Instead of using your parser in the app that consumes the file, use your app in the preprocessor and have it enmity standard JSON. (You'll have to start using a preprocessor.) Don't maintain both formats, only the original (that's how preprocessors should always be handled).

JSON's strength and it's very reason for existing is its simplicity and unambiguity.

sneak• 14 years ago

A thousand times this! Stop attempting to fragment a working standard for no benefit!

greghinch2• 14 years ago

(young) people seem to always want to find ways to skirt the "rules" that Crockford sets up (see also, jslint). Just stop. The rules he lays out are there to make your _long term_ programing life easier. Yes you may have to remember to use semicolons and quote things, but when you have to live with an application you've been working on for years, hopefully you'll learn to appreciate the fact that putting in a little extra time now saves you tons of headaches down the road.

aseemk15• 14 years ago

I'm actually a big fan of Douglas Crockford. And don't worry, my code uses semicolons. ;)

I'm not sure why you lump quoting keys with semicolons, though; his regular JS style guide doesn't discourage quoting object key (neither does JSLint).

ZenPsycho6• 14 years ago

there's a reason he requires it for JSON. it's that javascript has problems when you use a javascript reserved word as a key- for instance, if you were making an RPC api, and you had a json object {do:"methodname"} the javascript eval method would throw a fit. So rather than include the full list of javascript reserved words in a spec intended to be language agnostic, he just made quoting the key a requirement, which makes things vastly simpler.

tomjen3• 14 years ago

When was Crockford elected to set the rules for programmers and why wasn't I given a vote?

Oh, he was never elected? Why the fuck should I care about what one man thinks then?

mryan• 14 years ago

You should try relaxing a bit more, this seems like an incredibly silly thing to get worked up about. There are plenty of legitimate things in the world to get angry about - object serialisers are not one of them.

IMHO, aggressive comments that contain nothing but sarcasm are not welcome here. Let's try and keep the place a bit friendly (or at least grown-up and professional), even if we disagree. If you are going to be sarcastic, at least have the decency to make an interesting point at the same time.

> When was Crockford elected to set the rules for programmers and why wasn't I given a vote?

Crockford was obviously not "elected" in any official sense. However, he produced something that the programming language needed, and it very quickly became a standard. Have you ever used JSON in a project? Then you implicitly voted for JSON and, by extension, Crockford.

> Oh, he was never elected? Why the fuck should I care about what one man thinks then?

Because the rest of the industry has decided that JSON's benefits outweigh its downsides, and using it is a net positive. JSON is not perfect, but it solves an annoying problem rather elegantly.

greghinch2• 14 years ago

Well if you use JSON, you did vote, because he created the spec.

If you've never read his book or watched any of his talks, maybe you should check them out. He never says "this is how you have to do it". He says something to the effect of 1) JavaScript is a flawed language. 2) DC has used/helped develop it for a long time 3) he has figured out a lot of least bad solutions and habits to try and for to mitigate common problems. If you don't want to follow his suggestions fine, but don't be surprised when you run into the problems that he created them to help you avoid.

ZenPsycho6• 14 years ago

This entirely misses the point of JSON. if you want these features it's perfectly acceptable to use YAML instead of JSON.

kwamenum86• 14 years ago

I don't think any of these suggestions make JSON easier to read, generate, or consume. What problem are you setting out to solve?

Opening paragraph: "JSON is strict. Keys need to be quoted; strings can only be double-quoted; objects and arrays can't have trailing commas; and comments aren't allowed."

If this represents a comprehensive list of the problems JSON5 solves I think you'll have a hard time rallying people around your cause.

aseemk15• 14 years ago

Yes, I realize I didn't explain it as well in the README as I had in conversations and emails.

I find JSON tedious and error-prone to generate by hand. I also frequently wish I could document the data with comments.

Others have written about these same ideas, e.g. [1]; after feeling the frustration for over a year, I decided it might be productive to actually build a JS parser to get the conversation started.

My main use cases are Node/npm package.json files, and test case data.

[1] http://bolinfest.com/essays/json.html

ninjin• 14 years ago

> I find JSON tedious and error-prone to generate by hand. I also frequently wish I could document the data with comments.

I whole-heartedly agree with you on the tediousness, but JSON has this dubious double-nature that it is sometimes written by hand but mostly used as a protocol. As a protocol we want (and I would even argue need) this strictness since it makes day-to-day operations easier and parsers generally more reliable (how many times have we seen someone implement their own JSON parser? Now, what about XML? The first is a nice evening the latter is, well, not...).

As for package.json, I would argue the problem is their choice of serialisation rather than with JSON itself. Right tool for the wrong job. YAML would probably be a better pick.

arnarbi• 14 years ago

> I find JSON tedious and error-prone to generate by hand.

Why do you need to generate it by hand? Create object structures in whatever language you are working in and then simply serialize them to JSON.

drawkbox2• 14 years ago

JSON is great because it is simple basic cs types (string, numeric, boolean, object, list, date), a perfect middle ground between messaging and readability without going all XML/SOAP. And the unchanging simplicity is that way because it is so broadly accepted from Javascript to web services and beyond. It fits nicely into the system because of it's limitations but that also gives it standardized power.

Any changes to JSON should be fought til death. Use YAML or make a new format if you need more.

aseemk15• 14 years ago

Certainly. I'm not advocating for a change to JSON -- this is explicitly a new format. (The test cases use a .json5 extension to illustrate this.)

FWIW, I'm not adding any new data types or functionality, just making the syntax more human-friendly. It also continues to be a strict subset of ECMAScript, but v5 now.

drawkbox2• 14 years ago

Yes I was just talking in a general sense that JSON should not go all XML and mutate. In some cases I have added layers like this for instance in a C++ parser for game engines it allowed comments in the json but were stripped out at runtime on load and on send/receive from the services. But conforming to JSON that current browsers, libraries etc can use.

I see where you were going with it, ES5 support as a baseline is nice, also comments are nice but also bulky to a streamlined messaging format in most cases (config is usually yaml or some other format for this reason maybe).

I guess the naming is what maybe might cause some confusion and a good engineer makes things more simple not more complex, the simplicity of JSON is hard to achieve with such success but it is also its killer feature so that should be recognized. As long as we don't have to support 50 different 'modern JSON' formats then we are good. There are some good ones like BSON for binary JSON. I guess JSON5 works for your ES5 support.

I think here you have good reasons for your work but it goes back to the age old be 'conservative in what you send, liberal in what you accept'. That standards agreement makes everything work and to me being an engineer is taking complexity and simplifying always, bloat is bad. Here it isn't that but XML started out simple as well and became a verbose, bloated hell so people get touchy with JSON hehe.

catch23• 14 years ago

The name is definitely confusing though. You're creating a new format that existing JSON parsers cannot handle. In a way, it's like creating a new XML format called XML2 that uses parenthesis instead of angle brackets, and standard parsers like libxml will no longer be able to read.

Future newbie developers will wonder why their xml reader has problems reading document.xml2 without actually realizing the format is different from xml.

It's worse in your case since JSON5 almost looks like regular JSON, but if one were to use one of your special features, it would cause any normal JSON parser to break. The newbie developer might not be sharp enough to realize he needs a completely different parser.

wulczer2• 14 years ago

JSON does not define a date type.

wulczer2• 14 years ago

Ooops, I might have obsolete information. The grammar on json.org does not define a date type, but ES5 seems to define a Data.prototype.toJSON function. Oh well.

genericdave• 14 years ago

Seems like you're getting a lot of criticism for this, a lot of which I have to agree with. Writing valid JSON is extremely simple and unfortunately it's hard to see how this project would do anything besides promote bad practices for a near-universal format.

Suggestion: turn these criticisms to your favor. Make this into something that promotes good form. Perhaps by acting as a compiler that transforms your forgiving JSON into valid JSON. Make a command line utility. Emphasize how the output passes JSONlint or something. In other words, I'm suggesting that you remake this into a sort of coffeescript for JSON.

In any case, the path you are on isn't gonna garner much support. Change up your stride a bit and you could be on to something rather nice, though.

YmMot3• 14 years ago

I won't begrudge someone a fun little personal project, however (almost) no-one will ever adopt this. It removes too many benefits of JSON/introduces drawbacks for very little pay off.

Yes, JSON is a pain to edit/write by hand at times, but that's an editing problem and therefore should be solved in the editor. In any halfway decent editor it should be trivial to extend it with helpers for editing JSON. Perhaps have it convert .json files to YAML(etc) on opening, and back to .json on save etc.

This is a problem that plagues many technical people (present company included). You had a good instinct, but then took a left turn somewhere. "JSON is hard to hand edit...let's undo all the speed/compatibility progress we've made with JSON in the last ~8 years and introduce flaws in the protocol with a new non-standardized format...all so it's superficially easier to edit by hand."

aseemk15• 14 years ago

I'd love to understand what benefits this removes and what drawbacks this introduces, aside from the chicken-and-egg problem of having any new format. Apologies if I've missed that in the discussion so far.

You're definitely right that some of the hand-editing problems could in theory be solved by tools, but it unfortunately doesn't address documentation/comments.

I've heard the chorus on YAML though, and I'll definitely be looking into it. Thanks for the feedback!

YmMot3• 14 years ago

JSON is supported natively (i.e. fast and correctly) in all modern JS environments. Yes, at one point we used Crock's parser but that's the past. Likewise, it currently is only used as a stop-gap measure to begrudgingly support older browsers.

If we're going to undo years of progress we should do it for a very good reason.

JSON has wide support in many languages. Some of them have fast low level parsers. All that work has to be duplicated. Again, this just doesn't seem sufficient reason.

Right now if I want to throw together a Python script to crawl a bunch of NPM packages and read their package.json (let's say to put together a chart or collect statistics), I can easily.

If NPM switches to JSON5, we lose compatibility without a lot of duplicated effort. Again, there's just not enough benefits.

I consider comments a flaw, because they add little benefit. JSON is a data serialization/interchange format and there shouldn't be much need to comment on it, it should be largely self explanatory. If you really need to tag things with info, you can in JSON. Adding comments is to tempting for people to abuse.

However, if you really need lots of comments, there are many existing formats which might be better suited for your task. JSON is ultimately for machines. It has a benefit that humans can read/edit it easily...but if you have data that is intended more for humans and needs lots of meta information like a complicated config file, you should either create a front end for it or use a format better suited to that purpose.

When you consider that writing a plugin for your editor, or just a stand-alone translator would solve all of the problems this does, there's no reason to put up with all the drawbacks.

What it comes down to IMHO, is this isn't suitable to replace JSON and for other uses there are already existing formats that are better.

mitchellh• 14 years ago

While we're at it, I felt it necessary to launch HTML7: https://github.com/mitchellh/html7

aseemk15• 14 years ago

Haha, I actually found that hilarious. Nice work.

chris24• 14 years ago

The license was a nice touch.

tferris2• 14 years ago

Please rename your project or take it down.

People here gave you enough reasons.

veemjeem• 14 years ago

don't worry, next week I'll release JSON6 and nobody will remember JSON5 anymore.

augustflanagan• 14 years ago

I think that it's really a shame to see people slamming aseemk here. He had an idea, worked on it, and released it. That's fantastic. There are a few constructive comments here explaining why his idea might not actually be that great, but there are also quite few comments that seem out of place for HN.

jQueryIsAwesome• 14 years ago

I can't believe people are taking this seriously...

Lets write a python parser who accepts semicolons instead of indentation, just to fix when I forget that i am not writing Javascript; oh, and and make it accept the "function" keyword instead of def.

iamcarbon• 14 years ago

If only it were this easy to extended JavaScript with a new serialization format! If the serialized text can't be assigned to a JavaScript variable, it's not JavaScript notion.

I also tried something similar a few years back when I was sick of dealing with XML and coming to terms with the required quotes.

http://xjson.org/

While it's certainly prettier to have native literals and optional quotes, it's simply not worth giving up the interoperability with JavaScript.

DavidAbrams3• 14 years ago

Seems like quoted keys are a pain in the ass. That's the only useful change I see here.

pygorex• 14 years ago

I have to agree with most of the criticism in this thread. The lack of comments, strict quoting, no trailing slashes - these aren't problems to fix they are features.

As others have noted this is not JSON and is not necessarily an improvement to JSON - so calling it JSON5 is presumptuous.

Embrace the criticism. Instead of JSON5 call it NotJSON - a standard that breaks the JSON rules for a few narrow use cases - namely when you want a hand-written, commented JSON-like syntax.

MarkR42• 14 years ago

Congratulations, you've made something which looks superficially like JSON, but most documents won't parse in 99.9% of JSON parsers. Interoperability is lost, nothing is gained!

jwr• 14 years ago

I don't think making JSON more human-writable and readable is a goal worth pursuing.

JSON has its flaws, but is a surprisingly robust and elegant solution to many data interchange problems. But it is not (and I don't think it was ever intended to be) a language for human-edited configuration and data.

jahewson• 14 years ago

This reminds me of I guy I met who was trying to build a clone of Twitter, but with messages of unlimited length. In his view this made it objectively "better". He understood the what of the software, but not the why.

If you really want ES5 for writing package.json files, then you could save the time and effort by just using ES5:

    var _ = 

    { 
      some: 'object literal',
      // a comment
      num: 3,
    }
   
    fs.writeFileSync(path.basename(__filename, '.js') + '.json', JSON.stringify(_))

timrobinson• 14 years ago

> [IDEA: Allow octal and hexadecimal numbers.]

Who actually uses octal, except by accident?

sneeu• 14 years ago

The only time I ever use it is with `chmod`.

jimrandomh• 14 years ago

You know that something has gone very deeply wrong somewhere when people oppose your project because they are ideologically opposed to comments. In my own experience, finding out that many JSON parsers reject comments was a real WTF moment, and a deal breaker for my config-file application, so I ended up using JSON-plus-comments for it instead of JSON. At the same time, lack of support for trailing commas and unquoted member names have been a minor but persistent thorn in my side for no good reason.

The justification for not having comments in JSON is that in the great disaster that was XML, some projects would parse the comments and take them as semantically significant. However, the real problem there was that parser libraries would expose the comments, and that some generators would put important information only in comments. But I think that these mistakes are unlikely to be repeated, and that the proposed alternative - moving all comments into the markup, or eliminating them entirely - is just obviously worse.

One of the things that ruined the XML ecosystem was a persistent belief that XML was to be read and written by machines, combined with a reality in which it was mostly used for human-written config files, leading to a tolerance for awful syntax (like prefixing every single attribute with a namespace, and the ridiculous CDATA notation). I'm seeing the same thing with JSON: A significant fraction of its use is for human-written and human-read config files (which often desperately need comments) and people are pretending it's strictly a data interchange format that shouldn't be used for that.

It is sometimes said that JSON was discovered, rather than invented - that the syntax was already out there. So it is with JSON5: There is nothing new in this, it is simply return to JavaScript Object Notation and bringing in the rest of what Javascript has.

So, all you pooh-poohing ideologists: please seriously rethink whether disallowing comments, trailing commas, and quoteless member names is actually a good idea. Consider this in light of the fact that JSON is widely used, today, as a config-file language, and that {"--":"This is a comment"} is ridiculous enough that no one does it in practice, can't be inserted on any line, and invites consumers of your data to parse the comments.

aseemk15• 14 years ago

Thanks for stating so eloquently what I've been struggling to communicate. The heart of the matter to me really does boil down to the fact that JSON is meant for humans, too, not just machines. You nailed this sentiment.

(As a friend pointed out, "Do you not comment your code just because it's meant for machines?")

Show HN: JSON5 — modern JSON

Related Stories