One filter per page

jq starters, one filter per page

Twenty-seven filters for developers and CLI users who pipe jq off Stack Overflow and want the grammar: everything is a filter, .foo and the null that means missing, the ? that hushes errors, .[2] and slices, .[] as a fan-out, the pipe as an assembly line, the comma's two answers, {} as a blueprint, select as a gate that returns nothing, map, length and keys and has, add/any/all, empty versus null, error and try, string interpolation slots, Oniguruma test/match/capture, --arg versus --argjson, reduce and foreach, to_entries, the -n/-R/-s doors, one filter per page.

Steve Hodgkiss 7 filters

A diagram, the classic trap, and one filter to go try this week. That's a page.

jq starters, one filter per page

Twenty-seven filters for developers and CLI users who pipe jq off Stack Overflow and want the grammar: everything is a filter, .foo and the null that means missing, the ? that hushes errors, .[2] and slices, .[] as a fan-out, the pipe as an assembly line, the comma's two answers, {} as a blueprint, select as a gate that returns nothing, map, length and keys and has, add/any/all, empty versus null, error and try, string interpolation slots, Oniguruma test/match/capture, --arg versus --argjson, reduce and foreach, to_entries, the -n/-R/-s doors, one filter per page.

Set in Space Grotesk, Inter and JetBrains Mono (SIL Open Font License).

jq behaviour checked against the official jq manual (jqlang.org/manual, jq 1.8); every example filter verified against a live jq binary during the build. Regex behaviour is Oniguruma as the jq manual states it. The one practice claim, spaced practice beats cramming, is Cepeda et al. 2006, Psychological Bulletin, a meta-analysis of 317 experiments (PubMed 16719566). Teaching conventions (one rep a day, read the filter left to right) are named as conventions. jq is a project of the jq community (github.com/jqlang/jq); this book is an independent guide and is not affiliated with or endorsed by the jq project.

General information only, for practice and reference. Not legal, security, or professional advice; verify commands against your own systems and the current jq documentation before relying on them.

Your purchase is for personal use only. You do not have redistribution rights: please do not share, resell, or republish this book or its pages.

Edition 1.0 · stevehodgkiss.net

Contents

Everything is a filter

Input in, output out, always

Let's say you run jq . to make a wall of JSON readable and think that's all jq is. That dot is a whole program.

A jq program is a filter: one input, one or more outputs. The manual's first sentence. . is the identity filter, it hands its input through unchanged, and since jq pretty-prints by default, jq . validates and formats. Even 42 is a filter: any input, always 42 out. Hunting for the extract command is the beginner move; there is none, only filters you compose.

Everything is a filter. Even the dot.

TRY THIS WEEK

Pipe any API response through jq . today and read the pretty output once, end to end.

Object Identifier-Index .foo: when given an object, .foo produces the value at key foo if present, or null otherwise. .foo.bar chains. Per the jq manual, Basic filters.

jq starters · No. 02

Start here

.foo, the key lookup

Present, or null. Never a guess

Let's say .total comes back null in your report and you panic about the API. The key just wasn't there.

.foo gives the value at that key, or null if the key is missing, says the manual. Never an error, never a guess: .foo.bar chains the lookups. Treating that null as a value that exists is the quiet bug in a hundred scripts.

Missing key, null. That's the contract.

TRY THIS WEEK

Run jq '.foo' on {"foo": 42} and on {"notfoo": true} this week and see null, not an error.

The value iterator .[]: omitting the index returns all the elements of an array as separate results, or all the values of an object. It is a generator. .foo[] is equivalent to .foo | .[]. Per the jq manual, Array/Object Value Iterator.

jq starters · No. 03

Streams

.[] iterates

One array in, many values out

Let's say jq '.[]' prints your array's items one by one and you wonder where the brackets went. You just met the generator.

.[] produces every element as a separate result: [1,2,3] in, then 1, then 2, then 3, says the manual. On an object it yields every value. .foo[] is just .foo | .[]. Expecting one array back is why the next filter surprises you: it runs once per result.

.[] is a fan-out. One in, many out.

TRY THIS WEEK

Run jq '.[]' on [{"name":"JSON"},{"name":"XML"}] this week and count the separate results.

Part 3 of 5

Gates, rulers, juries

3

Choosing and measuring

select as a gate that returns nothing, map as the named idiom, and the measurers: length, keys, has and in, add, any and all.

In this part

01select(), the gate
02map(), the same move on every item

select(boolean): produces its input unchanged if the condition returns true, and produces no output otherwise. Useful for filtering lists: [1,2,3] | map(select(. >= 2)) gives [2,3]. Per the jq manual, select.

jq starters · No. 04

Choosing

select(), the gate

True passes. False vanishes

Let's say you filter a list and the failures come out as false, so you filter again. They didn't come out at all.

select passes its input through unchanged when the condition is true, and produces no output when it's false, the manual's words: nothing, not false, not null. The classic: [1,5,3,0,7] | map(select(. >= 2)) gives [5,3,7]. select on an array without .[] first tests the array itself and returns all of it, or none.

select returns nothing. That's the point.

TRY THIS WEEK

Run jq -c 'map(select(. >= 2))' on [1,5,3,0,7] this week. Then try it without map and explain the difference.

map(f) applies the filter f to each of the values of .[] in the input array or object and always outputs an array. map(f) is equivalent to [.[] | f]. Per the jq manual, map and map_values.

jq starters · No. 05

Choosing

map(), the same move on every item

Iterate, apply, collect

Let's say you write .[] | . * 2 and get a column of numbers when you wanted an array. map is that idiom with a name.

map(f) applies f to every value and hands back an array, always, says the manual: map(.*2) on [1,2,3] gives [2,4,6]. And the manual's own equivalence: [.[]|f]. map(select(...)) looks doubled until you read it as: keep what passes, collected.

map is .[] and a collect in one word.

TRY THIS WEEK

Rewrite one jq '.[] | select(...)' pipeline as 'map(select(...))' this week and diff the outputs.

The three input modes: --null-input / -n runs the filter once with null as input; --raw-input / -R passes each line as a string instead of parsing JSON; --slurp / -s reads the entire input into one array and runs the filter once. -R with -s passes the whole input as one string. Per the jq manual, Invoking jq.

jq starters · No. 06

Long filters

-n, -R, -s: the three doors

How input arrives at all

Let's say a log file has JSON per line and jq says parse error at line 1. Nothing was wrong with your filter; the input mode was wrong.

-n runs once on null. -R hands you each line as a string. -s slurps everything into one array and runs once, the manual's three definitions. Fighting a filter when the door is wrong is the classic hour lost.

Before the filter, choose the door: -n, -R, -s.

TRY THIS WEEK

printf 'a\nb\nc\n' | jq -R -s 'split("\n") | length' this week. Raw in, slurped, split.

The whole grammar in one filter: every page assembled into a single jq invocation that reads left to right: iterate, gate, build, collect, with --arg passing a value and -c compacting output. Every operator as the jq manual defines it.

jq starters · No. 07

The practice

The whole book in one filter

Every piece in its place

Let's say you want the one filter pinned above your desk. Here it is, and you built every piece.

jq -c --arg env prod '.items[] | select(.env == $env) | "\(.name): \(.count)"'. The comma never appears; the pipe carries; select gates; the slot interpolates; --arg arrives typed. Pasting this without naming each stage is where you started.

One line. You can read every stage of it.

TRY THIS WEEK

Write that filter from memory once this week against any array of objects. Check it stage by stage.

Index

-n, -R, -s: the three doors11

.[] iterates7

.foo, the key lookup6

Everything is a filter5

map(), the same move on every item10

select(), the gate9

The whole book in one filter12

jq starters, one filter per page

jq starters, one filter per page

Contents

The grammar

A jq program is a filter: it takes an input and produces an output. Even literals are filters. The identity filter . produces its input unchanged, so jq . pretty-prints and validates JSON. Per the jq manual's opening and the Identity section.

Everything is a filter

Object Identifier-Index .foo: when given an object, .foo produces the value at key foo if present, or null otherwise. .foo.bar chains. Per the jq manual, Basic filters.

.foo, the key lookup

The value iterator .[]: omitting the index returns all the elements of an array as separate results, or all the values of an object. It is a generator. .foo[] is equivalent to .foo | .[]. Per the jq manual, Array/Object Value Iterator.

.[] iterates

Choosing and measuring

select(boolean): produces its input unchanged if the condition returns true, and produces no output otherwise. Useful for filtering lists: [1,2,3] | map(select(. >= 2)) gives [2,3]. Per the jq manual, select.

select(), the gate

map(f) applies the filter f to each of the values of .[] in the input array or object and always outputs an array. map(f) is equivalent to [.[] | f]. Per the jq manual, map and map_values.

map(), the same move on every item

-n, -R, -s: the three doors

The whole grammar in one filter: every page assembled into a single jq invocation that reads left to right: iterate, gate, build, collect, with --arg passing a value and -c compacting output. Every operator as the jq manual defines it.

The whole book in one filter

Index