strace, one lesson per page
Thirty lessons for the tracer that watches the kernel boundary: what a syscall is and why the boundary beats guessing, launching under strace or attaching to a running pid, -f following forks, -e trace= filtering and the %file and %network sets, reading return values and errno patterns like ENOENT and EACCES, -c summary counts and -w wall clock, -T durations, -s string limits and -xx hex, and the ptrace mechanism underneath with its cost.
A diagram, the map, and one thing to try this week. That's a page.
strace, one lesson per page
Thirty lessons for the tracer that watches the kernel boundary: what a syscall is and why the boundary beats guessing, launching under strace or attaching to a running pid, -f following forks, -e trace= filtering and the %file and %network sets, reading return values and errno patterns like ENOENT and EACCES, -c summary counts and -w wall clock, -T durations, -s string limits and -xx hex, and the ptrace mechanism underneath with its cost.
Set in Space Grotesk, Inter and JetBrains Mono (SIL Open Font License).
Every fact in this book is as the official source states it, fetched and read during this build: the strace(1) manual at strace.io and at man7.org/linux/man-pages, the local strace 6.8 manual page shipped on this server plus its live command output observed directly, and the ptrace(2) manual at man7.org for the mechanism page; option availability across strace versions is flagged where it differs, and observed output shapes are cited as observed. Demand evidence from live command-line threads, reconfirmed at dispatch; no facts are sourced from tutorials, howtogeek, linuxize, Reddit or StackExchange. An independent guide, not affiliated with or endorsed by the strace developers.
Your purchase is for personal use only. You do not have redistribution rights: please do not share, resell, or republish this book or its pages.
© 2026 Steve Hodgkiss. All rights reserved. Personal use only; no redistribution rights.
Edition 1.0 · stevehodgkiss.net
Contents
Per the strace(1) manual: 'system calls and signals are events that happen at the user/kernel interface, a close examination of this boundary is very useful for bug isolation, sanity checking and attempting to capture race conditions.'
The kernel boundary
Everything a program does that matters, opening files, reading sockets, mapping memory, happens by asking the kernel. A system call is one such ask; a signal is the kernel tapping the program back.
The manual's insight: these asks are events at the user/kernel interface, and a close examination of this boundary is very useful for bug isolation, sanity checking and capturing race conditions.
Logs tell you what a program says it did. The boundary tells you what it actually asked for.
Run strace /bin/true once this week and just count the lines: every one is a kernel crossing before the program does any work.
First traces
Prefix a command with strace and read the stream from stderr, or attach to a running pid with -p and detach with a keyboard interrupt, leaving the process running; return values carry the verdict, -1 plus errno, and strings clip at 32 bytes unless -s widens them.
- 01Launch under strace
- 02Attach, don't restart
Per the strace(1) manual: 'The name of each system call, its arguments and its return value are printed on standard error or to the file specified with the -o option.' Observed with local strace 6.8: strace /bin/true ends with '+++ exited with 0 +++'.
Launch under strace
The simplest use is a prefix. strace command runs the command to completion with the trace streaming alongside, per the manual, on standard error, or to a file you name with -o.
That detail matters: the trace is not the program's output. On this server, strace 6.8 on /bin/true ends its stream with +++ exited with 0 +++, the exit recorded too.
Because the trace is stderr, a plain 2 redirect or -o file keeps it separate from the data you are testing.
Run strace /bin/echo hi this week and confirm the echo output and the trace arrive on different streams.
Per the strace(1) manual: '-p pid, --attach=pid: Attach to the process with the process ID pid and begin tracing. The trace may be terminated at any time by a keyboard interrupt signal (CTRL-C). strace will respond by detaching itself from the traced process(es) leaving it (them) to continue running.'
Attach, don't restart
You rarely get to restart the process that is misbehaving. Per the manual, -p pid attaches to a running process and begins tracing, and a keyboard interrupt ends the trace by detaching, leaving the process running.
Multiple -p options attach to several processes at once, and -p accepts a list: the manual notes -p "$(pidof PROG)" as supported syntax.
Attach to the thing that is wrong now. When you have seen enough, detach and the server never noticed.
Attach once this week: find a sleeping pid of your own, strace -p it for ten seconds, and detach with Ctrl-C.
Per the strace(1) manual: 'The default qualifier is trace... -e open means literally -e trace=open which in turn means trace only the open system call. By contrast, -e trace=!open means to trace every system call except open. In addition, the special values all and none have the obvious meanings.'
Trace one call
Unfiltered traces scroll. The -e expression narrows them: per the manual, -e open means -e trace=open, trace only that call, and -e trace=!open traces every call except open.
The special values all and none do what they say. Multiple names join in one comma list, and a leading question mark suppresses the error when nothing matches.
Two shapes cover most days: keep only the call you suspect, or drop the calls that drown it.
Cut one noisy trace this week with -e trace=!futex,mmap and see what was hiding underneath.
Per the strace(1) manual: '-c, --summary-only: Count time, calls, and errors for each system call and report a summary on program exit, suppressing the regular output. This attempts to show system time (CPU time spent running in the kernel) independent of wall clock time.'
Count, don't scroll
Before reading a single line, count them all. Per the manual, -c counts time, calls, and errors for each system call and reports a summary on exit, suppressing the regular output entirely.
The table is ranked, and on this server a plain /bin/true shows a few dozen calls led by mmap and openat: startup cost, visible as arithmetic.
The summary is the map. The lines are the territory. Read the map first.
Run strace -c on your slowest command this week and note the top row before touching anything else.
This book's plan, resting on the strace(1) manual: when a program misbehaves and the logs stay silent, go to the boundary first, because system calls and signals are 'events that happen at the user/kernel interface'.
One habit to keep
One habit survives every tool churn: when the logs are silent and the debugger is a step too far, go to the boundary first. Launch under strace, or attach with -p.
Then the ladder you have climbed: read the -1 lines for the errno, filter with -e and the named sets, count with -c, time with -T, and remember the stops cost the tracee speed.
The kernel cannot lie about what it was asked. Everything in this book is just ways of asking it.
Keep the ladder somewhere visible this week: attach or launch, read errnos, filter, count. One run per day and it sticks.