disk and io, one command per page
Twenty-eight lessons on disk and io, one command or file per page: lsblk's device tree, the major:minor pair and the -t topology columns; /proc/diskstats' seventeen counters, the read and write fields, in-flight and tick counts; iostat's -x windows, rate against bandwidth, await and r_await, %util with aqu-sz; iotop's per-process columns, Total against Current bandwidth, -o and -P; df's blocks and inodes beside du's device usage and apparent size; stat's blocks of 512 bytes, stat -f's filesystem arithmetic and the sysfs queue sizes; and fio measuring the same disk with sequential reads and random writes, one real command per page.
One command, one file, one real number. That's a page.
disk and io, one command per page
Twenty-eight lessons on disk and io, one command or file per page: lsblk's device tree, the major:minor pair and the -t topology columns; /proc/diskstats' seventeen counters, the read and write fields, in-flight and tick counts; iostat's -x windows, rate against bandwidth, await and r_await, %util with aqu-sz; iotop's per-process columns, Total against Current bandwidth, -o and -P; df's blocks and inodes beside du's device usage and apparent size; stat's blocks of 512 bytes, stat -f's filesystem arithmetic and the sysfs queue sizes; and fio measuring the same disk with sequential reads and random writes, one real command per page.
Set in Space Grotesk, Inter and JetBrains Mono (SIL Open Font License).
Every fact in this book is as the official source states it, fetched and read during this build: the Linux man-pages man7.org pages for lsblk(8) from util-linux, iostat(1) from sysstat, iotop(8), df(1), du(1), stat(1), stat(2) and inode(7) from GNU coreutils and the man-pages project; the kernel's own Documentation admin-guide/iostats.rst from kernel.org; and the fio project's own fio(1) manual from its source repository (fio 3.36; fio is not on man7.org). No facts are sourced from Reddit, Stack Overflow or vendor blogs. Every command and output pair on every page was run on this machine (util-linux 2.39.3, sysstat 12.6.1, iotop 0.6, fio 3.36, coreutils 9.4, kernel 6.8.0-147-generic) and confirmed before printing. An independent guide, not affiliated with or endorsed by the util-linux, sysstat, iotop, coreutils or fio projects, the Linux man-pages maintainers, or the Linux kernel developers.
Your purchase is for personal use only. You do not have redistribution rights: please do not share, resell, or republish this book or its pages.
© 2026 Steve Hodgkiss. All rights reserved. Personal use only; no redistribution rights.
Edition 1.0 · stevehodgkiss.net
Contents
Per Linux man-pages lsblk(8) (util-linux): lsblk reads the sysfs filesystem and udev db to gather information; by default, the command prints all block devices (except RAM disks) in a tree-like format; the tree-like output is enabled only if the NAME column is present in the output or when --tree is used; whenever possible, avoid using default outputs in your scripts and always explicitly define expected columns by using --output.
lsblk's one tree
Bare lsblk prints every block device except RAM disks as one tree. Ours: sda, 150G at the top, TYPE disk, then four partitions below it, TYPE part.
The mounts: sda1 carries /, sda16 /boot, sda15 /boot/efi. sda14 exists and mounts nothing.
Per lsblk(8) it reads sysfs and the udev db, and default output can change between versions: scripts should name their columns.
Run lsblk on your machine and say aloud which partition carries / and which carries boot.
Per kernel Documentation admin-guide/iostats.rst: the kernel exposes disk statistics via /proc/diskstats and /sys/block//stat; both files contain the same 17 statistics; in /proc/diskstats the fields are prefixed with the major and minor device numbers and the device name; all fields are cumulative, monotonic counters, except for field 9, which resets to zero as I/Os complete; the remaining fields reset at boot, on device reattachment or reinitialization, or when the underlying counter overflows. This machine's file printed lines for loop0 to loop7, then sda, sda1, sda14, sda15, sda16.
cat /proc/diskstats
The file under every disk tool: one line per device, opened by the major, minor, name you just met, then 17 counters.
Per the kernel's iostats.rst they are cumulative and monotonic, reset only at boot or device reinit. The quiet loop devices print zeros; sda's line is long.
iostat differences these lines over time; fio reads them after every run.
Run cat /proc/diskstats twice, a minute apart, and watch which numbers move.
Per Linux man-pages iostat(1): await is the average time in milliseconds for I/O requests issued to the device to be served, and it includes the time spent by the requests in queue and the time spent servicing them; r_await, w_await, d_await and f_await are the same measure for read, write, discard and flush requests; f_await can be twice as long because the block layer combines flush requests and executes at most one at a time. This machine's since-boot sda row read r_await 0.97, w_await 2.05, d_await 1.23, f_await 0.70.
await and r_await
await is the average milliseconds from issue to done, and the manual is exact: it includes time in queue plus time servicing.
Our since-boot split: reads 0.97 ms, writes 2.05, discards 1.23, flushes 0.70. Writes wait twice as long as reads here.
A rising await with flat rate is the classic queueing signal: requests are waiting longer, not arriving faster.
Run iostat -x 1 5 during a backup and watch w_await climb while the backup streams.
Per Linux man-pages iotop(8): iotop watches I/O usage information available in the Linux kernel, requires 2.6.20 or later, and displays a table of current I/O usage by processes or threads on the system; iotop displays columns for the I/O bandwidth read and written by each process or thread during the sampling period, the percentage of time the thread or process spent while swapping in and while waiting on I/O, and for each process its I/O priority (class/level); at least the CONFIG_TASK_DELAY_ACCT, CONFIG_TASK_IO_ACCOUNTING, CONFIG_TASKSTATS and CONFIG_VM_EVENT_COUNTERS options need to be enabled in the kernel build. This machine, with kernel task delay accounting switched on and a dd running, printed PID 3141342 be/4 root 0.00 B/s 144.79 M/s 0.00 % 0.00 % dd if=/dev/zero of=/tmp/diskio-dd.
iotop's columns
top, but for I/O. The columns: TID or PID, PRIO the I/O priority, USER, then DISK READ and DISK WRITE, bandwidth during the sampling period.
Our capture, with delay accounting on and a dd running: dd writing 144.79 M/s, the only row doing any I/O.
SWAPIN and IO, the percent-of-time columns, need kernel task delay accounting: the -P page covers that.
Run iotop -b -n 1 -o -P as root during a download and name your noisiest process.
Per Linux man-pages df(1) (GNU coreutils): df displays the amount of space available on the file system containing each file name argument, or on all currently mounted file systems if no file name is given; -h, --human-readable prints sizes in powers of 1024 (e.g., 1023M). This machine's df -h printed /dev/sda1 145G size, 40G used, 105G avail, 28 percent, mounted on /; /dev/sda16 881M, 117M, 703M, 15 percent, /boot; /dev/sda15 105M, 6.2M, 99M, 6 percent, /boot/efi; and tmpfs rows; 40 used plus 105 avail equals 145, and 40 of 145 is 27.6 percent.
df -h
One line per mounted filesystem: Size, Used, Avail, Use%. The root row: /dev/sda1 145G, 40G used, 105G avail, 28%.
The row adds up: 40 + 105 = 145, and 40 of 145 is 27.6 percent, which prints as 28.
The tmpfs rows live in memory, not on sda; -h scales in powers of 1024, per df(1).
Run df -h and add Used to Avail on your root row: the sum should be your Size.
Per Linux man-pages stat(1) (GNU coreutils) and inode(7): stat displays the file status; the stat(1) format %b is the number of blocks allocated and %B the size in bytes of each block reported by %b. This machine's stat /etc/hostname printed Size 11, Blocks 8, IO Block 4096, regular file, Device 8,1, Inode 636; lsblk printed sda1 as 8:1.
stat a file
One small file, all the facts: Size 11, Blocks 8, IO Block 4096, Device 8,1, Inode 636.
Eleven bytes of data, eight blocks allocated: small files still take whole blocks. And Device 8,1 is sda1, the same major:minor pair lsblk printed.
The next two pages chase the two block numbers: what a block is worth, and who sets the size.
Run stat on your smallest config file and find its Device pair in lsblk's output.
Per the fio project's own fio(1) manual (fio 3.36) and kernel Documentation admin-guide/iostats.rst: after a run fio prints disk stats for the devices used; this machine's read run printed sda: ios=6775/3, sectors=1734400/80, merge=0/5, ticks=2543/2, in_queue=2545, util=79.76%; per iostats.rst, ios, sectors and ticks are the same counters /proc/diskstats carries, sectors being field 3 and ticks field 10; 6775 read ios x 256 sectors per 128 KiB block = 1734400 sectors, exactly the printed count.
fio's disk stats
Under every run, fio prints the device's own tally: ios=6775/3 reads and writes, sectors=1734400/80, ticks, in_queue, util=79.76%.
The tie is exact: each 128 KiB block is 256 sectors, and 6775 x 256 = 1734400. ios, sectors, ticks are the /proc/diskstats counters you read on page five, snapshotted around the run.
fio watched the disk, not just the job: util 79.76 is what the device itself reported.
Run any fio job and check its sectors footer against block size times ios: it should land exactly.