jq - like sed for JSON data

http://stedolan.github.io/jq/

by gnosis • 13 years ago

132 29 13 years ago

stedolan.github.io

I've been using jq for a few months now while working with big GeoJSON blobs and it's a big help. The query language takes a bit of doing at first, but it's really pretty great once you learn it.

curl -s http://tile.openstreetmap.us/vectiles-skeletron/12/656/1582.... | jq '.features[].properties.name'

dc2447• 13 years ago

Personally jgrep seems a better choice here

curl foo/bar.json | jgrep -n -s 'property'

Or even better

curl foo/bar.json | jgrep 'status=500' -n -s 'path'

http://jgrep.org/

laumars• 13 years ago

Completely unrelated, but I love the typeface he's used on his site. In fact the whole site is laid out really nicely (pretty, yet none of the aesthetics gets in the way of usability. a balance that most sites fail to strike).

Anyhow, tangents aside, this looks a really handy tool. Thanks :)

revertts• 13 years ago

recs has this functionality, and _way_ more; worth giving a shot: https://github.com/benbernard/RecordStream

jcrites• 13 years ago

I'm a big fan of RecordStream. Everyone should definitely give it a look.

However, is it true to say that recs has this functionality? RecordStream is great with sequences of relatively simple JSON structures, where you want to count, group, collate, etc. the sequence. It looks like jq is better able to handle large, complex arbitrary JSON objects, pulling out deeply nested fields and transforming them. For example, how would you implement jq's "recurse" in RecordStream?

Now what would be really cool is using the two tools together :-)

doctorpangloss• 13 years ago

I like the concept, but how does it compare to using node + underscore?

Isn't the whole idea of JSON that it exactly serializes most JavaScript objects, and the "right way" to do stuff with it is to just use it like a normal object in your program?

Xorlev• 13 years ago

The spirit of the tool is to make JSON easier to work with in shell flows. There's plenty of times where you'll get a deluge of JSON and need to work with it (for debugging or whatever).

You may not have a program to use the JSON in.

mkehrt• 13 years ago

This assumes you're using JSON in Javascript. JSON is a pretty widely spread human-readable data format now outside of javascript, and it's nice to have good tools that aren't tied to a particular language.

nicpottier• 13 years ago

This is super cool, the language seems both powerful and easy enough to learn. Nicely done.

bsaul3• 13 years ago

Considering the amount of json around, wouldn't it be time to settle on a standard for :

- schema definition ( using something like typescript ? )

- data path ( similar to xpath )

- data transformation ( similar to xslt )

i'm pretty sure many people have done their things but i'm still waiting to see a tool like wsdl - based stub generators that would work on many json api.

ps : i know that things like jsonpath and wsdl2 exist, but i really feel like those technologies haven't caught widespread adoption.

haberman2• 13 years ago

I agree and I believe that the way to accomplish this is to use Protocol Buffer schemas.

Protocol Buffers and JSON have an extremely similar data model. The parts of them that people actually use are effectively identical. The actual differences are minor things like:

- JSON can have arrays of arrays; with Protocol Buffers you need a message in between.

- Protocol Buffers can represent binary data directly; with JSON you have to base64 encode it into a string.

- Protocol Buffers are more specific about the size of integer types (int64 and int32 are different); this allows for more efficient in-memory representations but doesn't matter so much on-the-wire.

I think the two are a match made in heaven.

bsaul3• 13 years ago

Yes, i also thought of protocol buffer right after sending that comment. Although i wonder : is there any way you could automatically generate a full service/model layer for, let's say angularjs or objective-c, based solely on protocol buffer "standard" ?

nly• 13 years ago

You're basically describing Piqi: http://piqi.org/

camus• 13 years ago

if you need all these , you should use XML. Crockford said he is not interested in any JSON standard extension, at all. not even to put comments in it. That's what XML is for.

Dont use JSON if you should use XML.

bsaul3• 13 years ago

I humbly disagree.

The reason i'm using JSON is for the fact that's it's fast to parse, compact, and lightweight, compared to XML. Now the reason for needing the xpath/xslt/xsd version of json is not principaly because it's a serialization format, but because it's a serialization format used in almost any modern web API. Experience shows that using JSON for web APIs is a good choice. Saying "you should've used XML" may have been relevant 5 years ago, but i think it's time to assume the fact that JSON is now so much used, that it needs the same kind of tools that xml has had almost from the start.

alexchamberlain• 13 years ago

I massively disagree. IMHO JSON is more readable, just as powerful and more concise.

quarterto2• 13 years ago

Nobody mentioned comments. Nothing the GP listed would in any more be an extension to JSON than HTML is to XML.

h2s• 13 years ago

Crockford's refusal to extend JSON is genius IMO. Totally agree with you: if the task you're trying to accomplish sounds like a fit for XML, then please use XML. JSON is what it is, and not everything needs to be gradually extended into six mutually incompatible versions of increasing bloat.

http://geekandpoke.typepad.com/geekandpoke/2012/03/thank-god...

kamaal• 13 years ago

And don't use XML if you have requirement where you need to query, insert, modify data pretty often.

Something like SQLite work wells for purposes like these.

belorn• 13 years ago

> schema definition ( using something like typescript ? )

There is a draft RFC called JSON Schema: core definitions and terminology (draft-zyp-json-schema-04).

> data path ( similar to xpath )

RFC 6901. (JavaScript Object Notation (JSON) Pointer)

guard-of-terra• 13 years ago

And for xml I highly recommend xmlstarlet.

query/transform xml documents from the command line.

evaneykelen• 13 years ago

For readers who are unfamiliar with sed: don't dimiss jq as yet another json pretty printer. From the jq tutorial "Data in jq is represented as streams of JSON values - every jq expression runs for each value in its input stream, and can produce any number of values to its output stream.".

I've played around with jq and it will definitely become a tool in my toolbox. Kudos to the author.

iso8859-12• 13 years ago

Previously on HN (141 days ago): https://news.ycombinator.com/item?id=4985250

quarterto2• 13 years ago

Clearly, this is new to enough people that it got to the front page. Merely linking to a previous submission adds nothing here; if you thing the discussion there was informative (clearly not the case here), then say so.

iso8859-12• 13 years ago

Yeah, I think the previous discussion was informative, it has alternatives and links to ever older relevant posts. Let the karma decide what HN likes. Like you said yourself: https://news.ycombinator.com/item?id=5664116

robmil• 13 years ago

Also available is the brilliant underscore-cli[1], which brings the functional programming of _.js to the command line.

There's an examples page, if you're struggling to see how it might be useful[2], but I find myself using 'extract', 'find', and 'pluck' on a daily basis.

[1] https://github.com/ddopson/underscore-cli

[2] https://github.com/ddopson/underscore-cli/blob/master/Exampl...

keenerd• 13 years ago

For a more unix-y version of this, check out Jshon.

http://kmkeen.com/jshon/

fabriceleal• 13 years ago

Some time ago I made this: https://github.com/fabriceleal/Jsoned using node.js.

It allows you to use Javascript expressions using a map/filter/reduce paradigm to edit the Json stream, like this:

echo "[1,2,3]" | jsoned -m "' _ * 2 '"

which will print "[2,4,6]"

you can even put the javascript code in files.

Check the wiki (incomplete) for documentation: https://github.com/fabriceleal/Jsoned/wiki/Usage

jq - like sed for JSON data

Related Stories