Working with JSON in bash using `jq`

jq is a powerful tool that lets you read, filter, and write JSON in bash

Cameron Nokes
Jul 16, 2018 · 5 min read
curl -s "http://api.icndb.com/jokes/random" \
| python -m json.tool \
| grep '\"joke\"' \
| cut -d ':' -f 2 \
| sed 's/"/\"/g'
curl -s "http://api.icndb.com/jokes/random" | jq '.value.joke'

Why not just use node.js when you need to deal with JSON?

Sometimes node.js is the right tool. For most automation tasks, I like to use bash whenever possible because it’s faster and even more portable (I can share a bash script with team members that don’t have node.js installed). To me, bash is more expressive and succinct for certain tasks than node is.

Install jq

jq isn’t a built-in command in any environment, so you have to install it. Run brew install jq on macOS. See jq’s install help page for how to install on other environments.

Basics of jq

jq works similarly to sed or awk — like a filter that you pipe to and extract values from. Also like sed or awk, it basically has it’s own domain specific language (DSL) for querying JSON. Luckily, it’s really intuitive (unlike awk 😜).

Get a property

Let’s say we have JSON that looks like this:

{ "foo": 123, "bar": 456 }
echo '{ "foo": 123, "bar": 456 }' | jq '.foo'
echo '{ "Version Number": "1.2.3" }' | jq '."Version Number"'

Iteration

Now let’s see how iteration works. The array or object value iterator operator, .[] , is what makes it possible.

echo '[1,2,3]' | jq '.[]'
echo '[ {"id": 1}, {"id": 2} ]' | jq '.[].id'
echo '{ "a": 1, "b": 2 }' | jq '.[]'
echo '["foo", "bar"]' | jq '.[1]'
echo '["foo", "bar"]' | jq '.[]' | touch

jq Functions

jq also has built-in “functions”. Returning to the previous object iteration example — let’s say we wanted get the keys of the object (not the values) as an array:

echo '{ "a": 1, "b": 2 }' | jq 'keys | .[]'
echo '[1,2,3]' | jq 'length'
jq -r '[(.dependencies, .devDependencies) | keys] | flatten | length' package.json

Let’s use it for real now

What if I wanted to audit my package.json dependencies and remove anything that’s not being used? Unused dependencies slow down npm installs for everyone and is just messy. I could manually grep usages of each dependency (via grep or in my IDE), but if you have a lot of dependencies that gets tedious fast, so let’s automate it.

See numbered comments below for an explanation.
  • -R means recursive, tells it to grep all matching files
  • --color colorizes the output
  • -n displays line numbers
  • -I {} defines the replacement string where the dependency string will get placed
  • -P 4 defines the concurrency, so 4 concurrent greps
  • we tell it to start a bash subshell where our grep_dep function is called with it’s args


Cameron Nokes

Javascript and front-end development how-tos

Cameron Nokes

Written by

Front-end developer and hamburger eater. Check out my screencasts: https://egghead.io/instructors/cameron-nokes

Cameron Nokes

Javascript and front-end development how-tos