Documentation for the JSON Lines text file format

https://jsonlines.org/

by gavmor • 2 years ago

93 88 2 years ago

jsonlines.org

larkinrichards• 2 years ago

Isn't this just ndjson? https://github.com/ndjson/ndjson-spec

Towaway69• 2 years ago

Yes -

Two terms for equivalent formats of line-delimited JSON are:
Newline delimited (NDJSON)[4] - The old name was Line delimited JSON (LDJSON).[5]
JSON lines (JSONL)[6]

https://en.wikipedia.org/wiki/JSON_streaming

rickette• 2 years ago

Pretty much, discussion: https://github.com/ndjson/ndjson.github.io/issues/1

Also when you're using this with GeoJSON there's https://stevage.github.io/ndgeojson/ which has an actual RFC (https://datatracker.ietf.org/doc/html/rfc8142)

zerocrates• 2 years ago

Which itself is based on an earlier RFC for not-specifically-Geo-JSON, RFC 7464. Both do things a little different than the others: they use the "record separator" character at the start of each line and actually split on that separator when parsing.

The GeoJSON one pretty much seems to exist just to hang an "application/geo+json-seq" media type registration off of. Part of me wants to say this really should have been more of a "all json subtypes are also json-seq subtypes" situation but maybe that's not really feasible with the standards/registration processes.

o11c• 2 years ago

It's strictly worse, actually.

ndjson specifies sane newline handling, since it works with terminators.

jsonlines works with separators and thus fails to detect truncated values. As a result, it can silently produce incorrect numeric values.

exyi• 2 years ago

This is only an issue with plain numbers, however. If you'd have the number in an object or array, you'd detect the truncation just as well. Since using jsonl for a plain list of numbers is... overkill, I'd say it's not an issue.

On the other hand, requiring line terminators in the standard would inevitably lead to incompatibility issues. Most software would accept unterminated files, because text libraries do; and so some files will not be terminated. Some applications do not line-terminate files even on Linux (hello VSCode), and it would be even more problematic on Windows

Kwpolska• 2 years ago

What’s the difference between terminators and separators here? The ndjson spec [0] doesn’t say anything like that, and it seems that ndjson and jsonlines are identical in what documents they accept.

[0] https://github.com/ndjson/ndjson-spec

simonw2• 2 years ago

This is such a convenient format to work with. It's especially great for streaming consumption without needing to load en entire document into memory at once - you can write a streaming parser just by consuming one line at a time and passing each line to a normal JSON parsing library.

mojuba• 2 years ago

It's also great for e.g. feeding search results to a client (the list-as-you-type kind), however JSONL doesn't specify how to end a stream. You will have to come up with your own higher-level protocol that allows having an EOD record.

throwaway143829• 2 years ago

Handling "end of data" isn't really something that any serialization format needs to specify, so I don't think this needs to be called out for JSONL specifically. e.g. in JS if you're using a ReadableStream you can just check whether a read returns .done==true. This is true regardless of the data you're reading, JSONL or otherwise

aidos• 2 years ago

Funny enough I’ve found myself just doing it naturally in cases where I was transferring a list of records. I’d still been using the json extension though feeling a bit off about it.

Recently (this year) I was flicking through a repo and saw a jsonl extension and immediately guessed what it would contain. It all just fits nicely.

oulipo2• 2 years ago

I've seen "slight improvements" on the format which says:

- ignore any line which is exactly "[" or "]" - ignore any trailing commas

This lets you consume your API content both as a JSONL stream, and as a JSON array

not sure if it's that useful though

zX41ZdbW2• 2 years ago

ClickHouse automatically detects variations of the JSONEachRow format:

- allowing optionally enclosing the data into an array;

- allowing comma as an optional separator between objects;

- allowing JSON objects without a newline separator;

- properly escaping Unicode newlines and / characters inside string literals for JavaScript compatibility;

- allowing binary data inside string literals;

jgalt2123• 2 years ago

It might be too early for my brain, but splitting on "\n" won't work if any of your json objects have newlines in the strings. Does this regex work? As I see it, the tricky one is matching a single string json object.

/(\]|\})\n|(.")\n/

https://www.debuggex.com/r/LaqAP5wmPFuRLNBO

simonw2• 2 years ago

New line characters aren't allowed in JSON strings - JSON uses \n for those.

throwanem2• 2 years ago

The JSON spec [1] explicitly forbids multiline strings. (Granted JSONL is already outside the spec, but the idea there being each single line contains valid JSON, encountering a multiline string in JSONL would if anything violate more expectations.)

[1] https://ecma-international.org/wp-content/uploads/ECMA-404_2...

enriquto2• 2 years ago

In a few years we will rediscover column-based, punctuation-free text files that can be parsed with single call to scanf(3).

saghm• 2 years ago

Maybe I'm reading too generously, but my sense is that the format described in the article is intended to be used for encoding and transferring data to be used in some other application rather than something where you can assume constraints like "no punctuation". I could imagine using the "JSON lines" format for things like encoding a large number of records to insert for test cases or as a way of sharing a test dataset with another developer; I'm not convinced that either of those would work well without any punctuation whatsoever, and even if they could, I suspect I'd still rather pay the cost of slower importing in order to make my testing code (and any manual reading/transformation outside of my code) easy to do without having to add some additional processing step to deal with the punctuation.

The best argument against this format to me is that it's possible to just use JSON itself for this by wrapping all of the values in an array. I'm not totally convinced that the quality of life improvements of not needing to add commas along with the single pair of square braces necessitate an entirely new file format, but realistically I think I'll just end up following whatever happens to be supported by the tooling I use for a given project. If "JSON lines" became widely used overnight, I wouldn't be bothered by it, but if I never end up seeing it supported by anything, I doubt I'll miss it.

Arch4852• 2 years ago

One particular downside to using an actual array instead of something like JSON lines is that you can't really stream the data.

If you have so much data that you need to stream it, you probably shouldn't be using JSON, though...

globular-toast• 2 years ago

The problem is text editor support. People don't trust binary data and want to see it in a text editor. ASCII has four characters reserved for separation so ridiculous formats like xSV where x is an in-band character are unnecessary. It's just the text editor support that is lacking. Even Emacs can't seem to do it well.

tracker1• 2 years ago

I used those characters in the latter 90s when sending query results to a web client. There was a set of parameters with ADO to seriously the data or to a string. It was easy enough to split/map it back in the client.

Using hidden frames for jsonp-like request and callback support, that was the fun stuff. Don't miss it.

Also don't miss XML or SOAP.

DonHopkins5• 2 years ago

And in a few years plus a nanosecond we will rediscover what a terrible, insecure, bug-prone, foot-gun api scanf(3) is, because of buffer overflow errors, lack of input validation, difficult error handling, clumsy newline handling, and format string vulnerabilities.

enriquto2• 2 years ago

I said that you can use scanf(3), not that you should. You can use "s.split()" in python, if that makes you feel safer.

fahhem• 2 years ago

I can't tell if this was created before or after all the Examples shown? It feels like people 'invented' this, then this website wrote down the implicit standard everyone already converged on.

anshargal• 2 years ago

There is also a similar thing for CSV files encoded as JSON arrays - csvj.org

remus2• 2 years ago

If the data can conveniently be split in to lines, I wonder if there is an advantage to this format over a single json list e.g.

    [
      {"name":"Bob"},
      {"name":"Alice"},
      {"name":"Alison"}
    ]

You're already working with json anyway so you could just stay in that world rather than mixing new lines and json.

est• 2 years ago

most JSON parser would have difficulty decoding an array with 1 million rows. For example in Python you could easily eat up all or RAM and swap and gets OOM'd

zipping1549• 2 years ago

If we're having milions of line of anything, just go with an actual db, no?

Expurple• 2 years ago

This doesn't work with streaming, e.g. JSON logs

krisoft• 2 years ago

The big benefit of JSON lines over a JSON array that you can easily append new records to your file by just adding new bytes at the end.

masklinn2• 2 years ago

More generally, the benefit of json lines is that it's trivially streamable, you don't need a streaming json serializer or parser, because each record can be serialized or deserialized by a common eager non-streaming parser.

remus2• 2 years ago

Ah yes, that is a big practical advantage, thanks for explaining.

beardyw2• 2 years ago

If you have a streaming parser it spits out what it's got and it waits for more data (or an end which may never come). Mostly you would be extracting what you needed from the JSON as it went along. Wondering if jq works that way?

beardyw2• 2 years ago

I agree. I think the objections result from JSON having it's roots in JavaScript as an object. I built a streaming JSON parser for a microcontroller with very limited memory. It wasn't too hard to do though it probably doesn't cover edge cases. But it can be done if you change your mindset.

jbverschoor• 2 years ago

Roots? JSON IS JavaScript. Its roots are:

  var data = eval(jsonString);

This is exactly why it sucks in many ways, but also why it's pretty great.

oulipo2• 2 years ago

well actually they are the same if you say (pseudocode)

``` const jsonLParser = (streamOfLines) => { for (line in streamOfLines) { if line.trim() === "[" || line.trim() === "]" -> skip if line.endswith(',') -> line = line.trimEnd(',') yield JSON.parse(line) } } ```

this allows your API to be consumed both by JSON parsers and JSONL parsers

ngalaiko• 2 years ago

can't get over the fact that this page uses

Tactile theme by Jason Long

and the format extension is .jsonl

DonHopkins5• 2 years ago

Does anyone find themselves accidentally mispronouncing Jason like "JSON"? It must be as annoying as mispronouncing Aaron like "A-A-Ron".

https://www.youtube.com/watch?v=Dd7FixvoKBw

throwanem2• 2 years ago

It's always been odd to me that "Jason" and "JSON" didn't end up homophonous. But as an A-A-Ron, if anyone named Jason ever quotes the Key & Peele sketch at me, I'll no longer need to think about how to retort!

iainmerrick• 2 years ago

Wait, isn't that exactly how you say JSON??

hden• 2 years ago

When will this get a proper IETF RFC?

larsga2• 2 years ago

From the page:

any help writing the RFC would be greatly appreciated (see issue).

I guess you can sign up if you want an RFC.

livrem• 2 years ago

That page could use an intro that explains what it is. Even the examples page begins with something about CSV instead of just a minimal jsonlines example.

I THINK that the format is like a JSON file with an implicit top-level list and the top-level (only?) commas replaced by newlines? But I do not see anywhere that is spelled out.

wruza2• 2 years ago

Can we PLEASE not screw this up in an actual standard and define a strict record terminator instead of separators?

Editors that remove or fail to add a final newline can go implement themselves.

yencabulator3• 2 years ago

NDJSON, which actually has a rudimentary spec, says

Each JSON text MUST conform to the [RFC8259] standard and MUST be written to the stream followed by the newline character \n (0x0A).

https://github.com/ndjson/ndjson-spec/

But yeah, these things should be merged and formalized. Being web-adjacent, how likely is that? What we're seeing here is the usual patterns:

- the less-useful newer thing with slightly better marketing wins eyeballs even when it's technically inferior

- an "ecosystem cornerstone" is someone's pet project from a prior employer and now neglected (https://github.com/ndjson/ndjson-spec/issues/35#issuecomment...)

- web things don't get RFCs written because web people are disconnected from old school internet people

- W3C adjacent people didn't get their RDF tuple world domination and industry standards don't matter to them

- even MDN is switching to paywalls instead of being a useful resource

larsga2• 2 years ago

MIME type may be application/jsonl

Don't do this. Don't encourage people to use a value that's not registered yet. Say "x-jsonl", and you're OK.

throw0101b• 2 years ago

MIME type may be application/jsonl, but this is not yet standardized; any help writing the RFC would be greatly appreciated (see issue[0]).

[0] https://github.com/wardi/jsonlines/issues/19

echoangle• 2 years ago

What’s the worst that can happen when using unregistered MIME types? Is it just aesthetics or does it make some functional difference?

ramses0• 2 years ago

Rule #2 is "illegal". JSON is actually "JavaScript OBJECT Notation", and raw integers are not "Objects". For compatibilities sake, I'd strongly encourage that NLD-JSON (New-Line Delimited JSON) contains valid JSON OBJECTS.

Some discussion here: https://stackoverflow.com/questions/17224536/why-is-one-numb...

...I am by no means the harbinger of all things JSON, but it's worth pointing out that this "spec" without addressing the issues that can arise from encouraging "bare" values (not wrapped in []/{}) is highly suspect.

palotasb• 2 years ago

Raw integers are also valid JSON according to the spec: https://www.json.org/json-en.html

json is defined as one element

element is defines as ws (whitespace), value, ws

value is defined as one of: object, array, string, number, "true", "false", "null"

Wrapping in {} to make an object or in [] to make a list (which is also not an object) is not necessary to make a valid JSON document. This is also discussed in the Stack Overflow question linked.

cprecioso• 2 years ago

JSON has always accepted "bare" values (try JSON.parse in the browser devtools). In the space, a valid json document is any of those "bare" values, or an array or object of those. All of the answers in your linked StackOverflow also mention this.

For example, the spec explainer at https://www.json.org/ (on the righ-hand), tells you that a `json` document is an `element`, which is a `value` surrounded by any or no whitespace. This `value` can be a literal, or an object or array.

For an (IMO) easier definition, you can check how JSON values are usually defined in TypeScript e.g. https://github.com/sindresorhus/type-fest/blob/main/source/b...

lifthrasiir• 2 years ago

That's only true for older version of specifications (which do NOT include json.org, however). The current RFC 8259 simply defines `json-text` as any allowed value.

slashdev• 2 years ago

A drawback of JSONL (and JSON) for me is that you can't have trailing commas. If the format allows for that, and ignores trailing commas at the end of the line you can stream and parse line-by-line, but you can also take any arbitrary group of lines, wrap them in [], again stripping the last comma (arghh!), and parse it with any standard JSON parser.

For example, Datadog doesn't ingest JSONL, so if you have JSONL log files you'll have to convert it to JSON first, you can't just use sendfile to upload your log in chunks, just wrapping it with [ and ] (which can be done with ordinary writes before and after sendfile.)

lencastre• 2 years ago

Isn’t this already “supported” by vscode?

SkyPuncher• 2 years ago

I don't understand what this solves. It's JSON format, with an alternative delimiter and lack of a wrapping array.

I guess the file extension ensures that JSON is not pretty printed.

serial_dev• 2 years ago

The point is that you don't need to read in the whole file before you can start working with the data.

This is great because it saves you memory (don't need to store the whole thing, your can store one line at a time depending on your specific issue), latency (don't need to wait for the whole file to be read and parsed, you start immediately after the first line), etc. Huge JSON files might be also too big to load into memory. You can read one line at a time, you can stream your lines, process your input across multiple computers etc.

And, as it's just json lines, you also don't need to invent a new notation, specialized tools, etc.

emadda• 2 years ago

Are there any neat log viewers that allow you to filter and tail logs in jsonl format?

I have many systems that log in this format but are quite hard to visually scan during development. I typically create ad hoc filters using jq, rg or a js script.

jurschreuder• 2 years ago

We always use:

<origin_id> <timestamp> <event_type> {payload_json} \n

This way you know what json parser to use for which line.

origin_id and timestamp is just something we always use with our EventStream

In the database it also has one iterator per origin_id, and within an origin one iterator per event_type.

This way you can listen to both an origin_id and a specific event_type while making sure you can batch fetch any missing data if the iterators skip a beat.

foofie2• 2 years ago

Can anyone explain to me what's the selling point of this format?

jlas• 2 years ago

It's useful for streaming data where you want to pass small chunks of data incrementally. JSON by itself isn't streamable (you need the entire blob to decode it). But if we just pass smaller json-encoded chunks of data and separate them with a newline, we can make a streaming-compatible JSON protocol.

foofie2• 2 years ago

It's useful for streaming data where you want to pass small chunks of data incrementally. JSON by itself isn't streamable (you need the entire blob to decode it).

What's wrong with JSON patch?

https://jsonpatch.com/

rubatuga• 2 years ago

This is a good time to mention another text based data format based on readability: NestedText. You can hand edit and then pipe valid NestedText into Unix programs, which makes it useful for portability. It only supports the string type, but this can be desirable for some applications.

Find out more at https://nestedtext.org/en/stable/

kazinator• 2 years ago

This is useful because it gives a low-level data link framing by which a processor can recognize JSON objects without understanding JSON.

A processor that only understands the data link format (lines) can do useful things like merge or interleave two or more such streams, delete the n-th record given n, and such.

Though, I wouldn't call it a vista of limitless possibilities or anything of the sort. :) As soon as you have to crack the JSON to do anything interesting, the line orientation becomes more or less moot.

Documentation for the JSON Lines text file format

Related Stories