Jq - a lightweight and flexible command-line JSON processor

http://stedolan.github.com/jq/

by tchalla • 14 years ago

143 36 14 years ago

stedolan.github.com

alexchamberlain3• 14 years ago

> You can download a single binary, scp it to a far away machine, and expect it to work.

Not if my machine is x86 and the server is SPARC.

Cool utility though.

evincarofautumn3• 14 years ago

Git clone and make, then. As long as you have a C compiler. And if you don’t, you probably aren’t using JSON.

hnriot• 14 years ago

Actually I bet there are a ton of people using JSON that don't have a c compiler. The technologies are spaces by several decades and I would imagine the majority of JSON people are using much higher level languages than c.

If I want to munge JSON I find it easier to do so in python, just convert it back to data structures and manipulate them and serialized it back to JSON.

evincarofautumn3• 14 years ago

We periodically have to munge JSON at work—will definitely be trying this out.

nekedorff• 14 years ago

Very nice tool! I'm getting syntax errors from the Twitter examples though (and everytime I try to use underscores):

<edit>Created an issue on Github</edit>

  curl 'http://search.twitter.com/search.json?q=json&rpp=5&include_entities=true | jq '.results[] | {from_user, text, urls: [.entities.urls[].url]}' 

  error: Invalid character
  .results[] | {from_user, text, urls: [.entities.urls[].url]}
                    ^
  error: syntax error, unexpected IDENT, expecting '}'
  .results[] | {from_user, text, urls: [.entities.urls[].url]}
                    ^^^^
  2 compile errors
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
  100  5970  100  5970    0     0  11969      0 --:--:-- --:--:-- --:--:-- 23050

philip_roberts• 14 years ago

Same issue here.

alexchamberlain3• 14 years ago

Post it here...

https://github.com/stedolan/jq/issues

stedolan4• 14 years ago

(author here) That bug was fixed a while ago, but I forgot to upload new binaries. Try again, they should be working now.

Draiken• 14 years ago

Very nice tool, congrats!

This is something everyone ends up needing at least one time, and we always end up using a language specific tool.

Nothing beats the command line piping

nsp• 14 years ago

D. DQ,-"l

metabrew• 14 years ago

See also, jsawk: https://github.com/micha/jsawk

tmcw• 14 years ago

See also, underscore-cli https://github.com/ddopson/underscore-cli

zootm• 14 years ago

Also see also RecordStream: https://github.com/benbernard/RecordStream

Someone mentioned jsawk in a sibling comment, too.

manveru• 14 years ago

Also see http://www.kmkeen.com/jshon/

almog2• 14 years ago

Got the following compile error when running make (Lion + XCode 4.5.1):

  bison -W -d parser.y -v --report-file=parser.gen.info -o
  parser.gen.c
  bison: invalid option -- W
  Try `bison --help' for more information.
  make: *** [parser.gen.c] Error 1

stedolan4• 14 years ago

(author here) I haven't tried to build on a mac in a while, bison there seems to support fewer options (must be an older version).

For now, I've just checked in the autogenerated parser, so that bison won't have to run when you build master. git-pull and try again :)

almog2• 14 years ago

Works great now, thank you:)

atdt• 14 years ago

It'd be great if the help text contained some examples rather than simply a pointer to the website, and if you added a man page. When you're in the middle of a terminal session, switching to a browser can be really disruptive.

Also, if you tag releases with a version number and make them available as downloads on GitHub, you'll have an easier time getting Jq into Homebrew, the OS X package manager. I wrote a formula for you, but AFAIK you'll need to add in a real version number before submitting a pull request to Homebrew: https://gist.github.com/3928074

alexchamberlain3• 14 years ago

Arch PKGBUILD would be much easier with a version number too...

stedolan4• 14 years ago

They exist now!

littlemerman• 14 years ago

This is awesome. Thank you.

ruggeri• 14 years ago

Thanks; very nicely done. :-)

xargs12• 14 years ago

Built for Windows using cygwin, works a treat.

MatthewPhillips• 14 years ago

Builds with MinGW as well, although I did have to manually edit a header file because I couldn't get sed to work properly.

055static2• 14 years ago

Wow, this compiles fast. And the Flex and Bison files are there. This seems hackable. Nice work!

My only questions are (they are always the same with any purported sed/awk "replacement"):

1. what was the problem you were trying to solve where sed and awk failed you, and

2. does this program operate line by line or does it read entire files into memory?

I had to deal with some JSON a while ago and threw together some sed like this just so I could read it:

    sed '
    s/,/&\
    /g;
    /^$/d;
    s/^[{][^}]/\
    &/g; 
    /\"/s/,/<##eol##>/g;
    s/ *//;
    ' |tr '\012' '\040' |sed '
    s/<##eol##>/\
    /g;
    s/\[/\
    &\
    /;
    s/\]/\
    &\
    /;
    s/,  /, /g;
    '

But any difficuly I have dealing with JSON I attribute to the pervasive use of JSON, not sed.

stedolan4• 14 years ago

Thanks!

1. I think your example shows sed/awk's failings with JSON data :) I don't want to write a JSON parser by hand every time I want to pull a field out of an object, and parsing recursive structures with regexes is never a good plan.

2. It reads JSON items from stdin into memory, one at a time. So if the input is a single giant JSON value, it all goes into memory, but if it's a series of whitespace-separated values they'll be processed sequentially.

It's cat-friendly: if you do

    cat a | jq foo; cat b | jq foo

then it's the same as doing

    cat a b | jq foo

055static2• 14 years ago

1. But those are general statements. Opinions. What I mean is give me a specific case. A specific example, a specific block of JSON and a specific task. Once I have that, then I can ask myself "Is this something I would ever need to do or that I have to do on a regular basis?"

Sometimes I need to write one-off filters. There is just no getting around it. I have to choose a utility that gives maximum flexibility and is not too verbose; I don't like to type. Lots of people like Perl, and other similar scripting languages for writing one-off filters. But Perl, _out of the box_, is not a line-by-line filter. It's unlike sed/awk; it needs more memory. That brings us to #2.

2. If I understand correctly, jq is reading the entire JSON block into memory. This is what separates your program and so many other sed/awk "replacements" from sed and awk, the programs they purport to "replace". sed/awk don't read entire files into memory, they operate line-by-line and use a reasonably small buffer. Any sed/awk "replacement" would have to match that functionality. Given that sed/awk don't read in an entire structure (JSON, XML, etc.) before processing it, they are ideal for memory constrained environments. (As long as you don't overfill their buffers, which rarely happens in my experience.)

Anyway, so far I like this program. Best JSON filter I've seen yet (because I can hack on the lexer and parser you provided).

Well done.

vhost-• 14 years ago

I vote for a man page!

codereflection• 14 years ago

http://stedolan.github.com/jq/manual/

bryogenic• 14 years ago

When doing the tasks this seems to be targeting, Jansson has worked well for me.

http://www.digip.org/jansson/

trustfundbaby• 14 years ago

FYI if you have python installed (and most do) you can just do

curl 'http://search.twitter.com/search.json?q=json&rpp=5&i... | python -mjson.tool

draegtun• 14 years ago

Also see App::PipeFilter - https://metacpan.org/module/App::PipeFilter

The module is a framework/collection of shell pipelines which include JSON processing:

  curl -s 'http://api.duckduckgo.com/?q=poe&o=json' |
  jsonpath -o '$..Topics.*.FirstURL' -o '$..Topics.*.Text' |
  grep -i perl |
  jmap -i col0 -o url -i col1 -o title |
  json2yaml

Output:

  title: Perl Object Environment, a library for event driven multitasking for the Perl programming language
  url: http://duckduckgo.com/Perl_Object_Environment

Other structured data can be added to this module (Looks like Ingy maybe adding YAML - https://github.com/ingydotnet/app-pipefilter)

ohaal• 14 years ago

See also jsontool, http://trentm.com/json/

Jq - a lightweight and flexible command-line JSON processor

Related Stories