Jq: A lightweight and flexible command-line JSON processor

http://stedolan.github.io/jq/#

by wtetzner • 12 years ago

229 55 12 years ago

stedolan.github.io

edgarbob• 12 years ago

Does anyone know of a similar tool for XML? In particular, it would be cool to have a useful way to run xpath, and maybe xquery, from the command line. The tools I've found have unfortunate interfaces, e.g. they won't read XML from stdin.

gav• 12 years ago

Have you tried XMLStarlet[1]? It's my go-to for command line XML processing.

[1] http://xmlstar.sourceforge.net/

unhammer• 12 years ago

Even with libxml's xmllint you can do some xpath stuff: http://wiki.apertium.org/wiki/Xml_grep

cgrubb• 12 years ago

You might check out these:

http://www.w3.org/Tools/HTML-XML-utils/

However, I just use a Ruby script which loads the Nokogiri library:

https://github.com/clarkgrubb/data-tools/blob/master/src/dom...

https://github.com/clarkgrubb/data-tools/blob/master/doc/dom...

jeroenjanssens• 12 years ago

I'd just use xml2json and then apply jq. Usually gets the job done. The benefit of that is that you only need to learn jq.

See this blog post I wrote a while ago for an example: http://jeroenjanssens.com/2013/09/19/seven-command-line-tool...

cryptonector9• 12 years ago

Most of the JSON conversions from XML get it all wrong though. XML is not really an object (dict, hash) based meta-schema. It's an array-based meta-schema. XML->JSON conversions should look something like this:

    [ {"tag":"something", "attributes": { ... }, "nodes": [ ...] }, ... ]

with nodes being objects with one key to indicate if the node is a text node or an element, and if a text node then a value, and so on.

bartc• 12 years ago

xmllint - http://xmlsoft.org/xmllint.html

A_COMPUTER• 12 years ago

I am not a huge fan of PHP, but you can do some useful stuff to XML using the PHP command line tool, provided you have the right libraries installed. I have run into situations where the package manager on a machine I'm using doesn't have an up to date xmlstartlet or xmllint, and have tidily gotten around this problem using PHP.

facorreia• 12 years ago

This can help Docker automation by processing the output of `docker inspect`.

Zolomon• 12 years ago

They use this tool extensively at Spotify. I saw a presentation by https://twitter.com/jooon, when I was over there in September last year.

It is a really cool tool. Especially since they use JSON internally for almost all of their data representations.

Jq + Percol[1] is kind of cool.

[1] https://github.com/mooz/percol - was on HN a few days ago.

X42• 12 years ago

If percol is based on aut-fu, then it's probably not that good. The screenshot shows auto-fu, I know that zsh-plugin, but it's really messy and does very aggressive auto-completion (erasing your previous input). I'd rather use fish-shell instead.

ithkuil• 12 years ago

no percol is not shell plugin, but a simple cmdline util used in a pipeline, I suggest you take a look. I understand that animation on the website is a little bit confusing though.

cryptonector9• 12 years ago

I was also thinking of Xiki (http://xiki.org/) + jq.

A Unix-like shell with JSON for object serialization could approximate the PowerShell, no? Think of ksh93's awful compound variables... done right.

cryptonector9• 12 years ago

Is this the link to that presentation http://www.slideshare.net/jooon ?

otikik• 12 years ago

I just use it just as a pretty-printer for the console (jq '.') , and it does the job very well.

simonw• 12 years ago

This is a really neat tool.

If you just need pretty printing and don't have jq installed, you can use the python command line with the built in JSON module:

wget http://reddit.com/.json | python -mjson.tool

jamiesonbecker• 12 years ago

# (fixed) output to stdout:

wget -O- http://reddit.com/.json | python -mjson.tool

rev087• 12 years ago

I've been using this for a while, and it's wonderful to parse the output from API calls requested via curl, as well as quickly checking a specific package.json field from the npm output. The --json flag is handy for the latter.

mathieuruellan• 12 years ago

I'm using this tool with curl in scripts calling AWS api. Looking for an id of an object (security group, autoscaling group etc.) with its name and referencing it afterward in another command.

pavel_lishin2• 12 years ago

Hey, me too! It's pretty awesome, although the documentation/examples aren't as great as they could be - it took me some time to figure out how to properly parse arrays, and I'm still not sure how to actually do what I want to do (I want to convert an array of two instance objects into an array with just two instance IDs.

mh-• 12 years ago

    % aws ec2 describe-instances --instance-ids '["i-3026a249","i-28739551"]' \
    | jq '[.Reservations[].Instances[].InstanceId]'
    [
      "i-28739551",
      "i-3026a249"
    ]

gatehouse• 12 years ago

I've been using it for the exact same thing. One difficulty that I had was that you need to use the raw option, "jq -r" for string output without quotes, but I was looking at the query language documentation and not finding it.

EDIT: here is one I used. Given VOLUME_ID, gets the most recent snapshot-id.

    SNAPSHOT_ID=$(\
    aws ec2 describe-snapshots --owner-ids xxxxxxxxxxxx --output json | 
      jq -r "
        .Snapshots | sort_by(.StartTime) | reverse[] |
        select(.VolumeId==\"$VOLUME_ID\" and .State==\"completed\") |
        .SnapshotId
      ")

EDIT2: now that I'm looking at it, I think that I expected it to work with multiple snapshots but I'm not sure that I tested it.

mentat• 12 years ago

I've been using this to transform the output of ElasticBeanstalk environment queries into things I can feed into environment updates since moving between versions is a PITA. Should post that code sometime...

skywhopper• 12 years ago

I've been using this tool to build a bash library for a JSON/REST API and I can attest to its usefulness. Filter huge data sets down to just the fields you want, mutate, select, search, filter. Most everything I've wanted to do this can handle. Plus it's fast and dead simple to install. Kudos to the devs for this fantastic tool that I hope will soon be part of the default CL toolset across the board.

Tyr42• 12 years ago

Here's a cute little jq script I used to munge some json around.

https://gist.github.com/tbelaire/d00cb77fb0db7b38bed2

reinier_s4g• 12 years ago

Awesome stuff, thanks! Will give it a try really soon.

slapresta3• 12 years ago

I can't stress how useful this is. I use it to process AWS SQS messages for debugging. Incredibly powerful tool.

I have also used it to validate sudokus, because I can.

https://github.com/slapresta/jq-sudoku

JoshTheGeek• 12 years ago

underscore-cli: a similar tool, in JS (node)

https://news.ycombinator.com/item?id=7896054

kolev4• 12 years ago

I've been using it for over a year - especially with AWS CLI, but, I'm a little displeased that it only has the query feature (duh) and does not have a little more like merging JSONs (I think it only has appending). Another issue is that I wasn't able to find release notes for v1.4 - what's new since v1.3? There other similar tools, but they require node, Python packages, etc. jq really is installed with one command and no dependencies.

lugg2• 12 years ago

You should be able to do some pipe fu and carry out some form of merging via bash. I mostly use it for making our output more readable or producing CSV style reports with the filtered data. Its filters are probably the biggest feature, at least in my mind.

kolev4• 12 years ago

Sorry for not being clear. I meant a merge in the form of {"a": 1, "b": 2, "c": 3} merged with {"b": 20} returning {"a": 1, "b": 20, "c": 3}. Like Python's update() or PHP's array_replace_recursive() although PHP's a little richer.

cryptonector9• 12 years ago

Yeah, I'm a schmuck, I forgot to add release notes :(

I've been thinking I need to do a 1.4.1 release just for that!

(Plus we're getting some cool new features suddenly. @wtlangford (on github) is working on regexps, for example. Maybe a 1.4.1 so soon would be justified.)

kolev4• 12 years ago

No, you're anything, but a schmuck! Thanks for the great project!

glesica• 12 years ago

jq is written in portable C, and it has zero runtime dependencies.

Thank you! When I clicked on the link I was fully expecting it to be written in Node and I was going to cry.

Xavi• 12 years ago

Why is having node as a dependency such a bad thing? No snark. Genuinely curious.

aendruk• 12 years ago

The subsequent sentence suggests that the benefit is convenience of impromptu deployment:

You can download a single binary, scp it to a far away machine, and expect it to work.

ajtaylor• 12 years ago

I don't think that node is bad, but it's simply another dependency you have to worry about. Most devs likely won't have node pre-installed. Having a no-dependency binary is a great thing.

cryptonector9• 12 years ago

Node is great, and a port of jq to node would be awesome (someone once hinted at such a port, but I've lost track of it).

But jq also has libjq, a C library. And jq uses a copy-on-write, reference counted representation of JSON values, which is, for example, inherently thread-safe (though jq isn't using atomic operations for refcount management yet). The library is easy to use and powerful.

Ultimately the main thing I love about jq (and why I contribute to it) is the jq language itself. It reminds me of my one-time favorite, Icon. But in a world where C is still the champion of systems programming (until Rust takes over?), it's real handy to have a C JSON library _and_ a functional DSL that's easy to invoke from C.

bsg75• 12 years ago

Because if you are not developing in node, and don't have any other node based software, yet another runtime, its package manager(s), and dependencies are unwanted additions to a system.

chrismonsanto• 12 years ago

This is a great tool. I've longed for an equivalent for YAML.

lugg2• 12 years ago

Been using jq for a while now. Probably since the last hn post. My biggest gripe with jq is the name and asking for help on stack overflow - the jq tag is an alias to the jquery tag. Someone with tag edit level rep should really fix that.

yincrash2• 12 years ago

You can probably make a post on meta stack about it

cryptonector9• 12 years ago

http://meta.stackoverflow.com/search?q=jq

davidbanham• 12 years ago

This script drops you into a coffescript REPL with the json loaded into memory under the variable j.

I find it nice to be able to loop the data, define new subsets, filter it with functions, etc.

It was a 10 minute slap-job to get it together, so it does barf on particularly large datasets. The obvious reason is that it serialises the JSON into an argument for a child process. Shouldn't be too hard to resolve if anyone feels like it.

https://github.com/davidbanham/dotfiles/commit/0ea950373604e...

minaguib• 12 years ago

If you get and build latest from master, you get the --sort-keys commandline option

And just like that, you have an amazing base for diffing json documents of arbitrary complexity:

    vimdiff <( cat doc1 | jq --sort-keys . ) <( cat doc2 | jq --sort-keys . )

cryptonector9• 12 years ago

It's in 1.4!

samuell• 12 years ago

Cool, just grabbed it with sudo apt-get install jq

Jq: A lightweight and flexible command-line JSON processor

Related Stories