network path debugging, one hop per page
Thirty lessons on network path debugging, one tool per page: reading ping's rtt line, TTL budgets and loss percentages, shaping runs with -c -W -w and -s; traceroute's ttl trick, hop lines, what a star really means and the -m -q -w dials; probing in udp, icmp and tcp dialects; mtr's per-hop Loss%, the Best Avg Wrst StDev columns and report mode; ss listeners, states, peers and queues beside netstat's retirement; the default route, ip route get and longest prefix wins; path MTU with the 1472 sum, the DF bit and tracepath; DNS versus path with the isolation order; and two real diagnostic runs, the slow page and the unreachable host, with one real command to run per page.
One tool, one command, one real output. That's a page.
network path debugging, one hop per page
Thirty lessons on network path debugging, one tool per page: reading ping's rtt line, TTL budgets and loss percentages, shaping runs with -c -W -w and -s; traceroute's ttl trick, hop lines, what a star really means and the -m -q -w dials; probing in udp, icmp and tcp dialects; mtr's per-hop Loss%, the Best Avg Wrst StDev columns and report mode; ss listeners, states, peers and queues beside netstat's retirement; the default route, ip route get and longest prefix wins; path MTU with the 1472 sum, the DF bit and tracepath; DNS versus path with the isolation order; and two real diagnostic runs, the slow page and the unreachable host, with one real command to run per page.
Set in Space Grotesk, Inter and JetBrains Mono (SIL Open Font License).
Every fact in this book is as the official source states it, fetched and read during this build: the Linux man-pages man7.org pages for ping(8) from iputils, traceroute(8), ss(8), ip-route(8), tracepath(8) and netstat(8) from iproute2 and net-tools, and ip(7); the mtr 0.95 package man page; and RFC 792 (Internet Control Message Protocol), RFC 1191 (Path MTU Discovery), RFC 1122 (Requirements for Internet Hosts) and RFC 2151 (A Primer on Internet and TCP/IP Tool Utilization). No facts are sourced from Reddit, Stack Overflow or vendor blogs. Every command and output pair on every page was run on this machine and confirmed before printing: iputils 20240117, traceroute 2.1.5, mtr 0.95, iproute2 6.1.0, net-tools 2.10. An independent guide, not affiliated with or endorsed by the Linux man-pages maintainers, the iputils, iproute2, traceroute or mtr projects, or the IETF.
Your purchase is for personal use only. You do not have redistribution rights: please do not share, resell, or republish this book or its pages.
© 2026 Steve Hodgkiss. All rights reserved. Personal use only; no redistribution rights.
Edition 1.0 · stevehodgkiss.net
Contents
Per Linux man-pages ping(8) (iputils): ping sends ICMP ECHO_REQUEST datagrams and prints round-trip time and packet loss statistics, closing with minimum/average/maximum/mdev round-trip numbers; mdev is the population standard deviation, essentially an average of how far each ping RTT is from the mean RTT. Confirmed on iputils 20240117.
ping and RTT
ping asks one question over and over: is there a round trip, and how long does it take? Each ICMP ECHO_REQUEST goes out, and each reply comes back stamped with the round-trip time, the RTT.
Our four probes to 1.1.1.1 returned in 4.75, 2.67, 1.92 and 1.94 ms, and the summary states them compactly: min/avg/max/mdev = 1.919/2.820/4.746/1.152 ms.
mdev is the spread: 1.152 ms says the path was steady, not just fast.
Run ping -c 4 1.1.1.1, write down the four rtt numbers from the summary line, and say in one sentence what the mdev tells you about steadiness.
traceroute
Per Linux man-pages traceroute(8): traceroute utilizes the IP time to live field, launching probes with a small ttl and eliciting an ICMP TIME_EXCEEDED response from each gateway, from ttl 1 up to a default maximum of 30 hops; each line prints the ttl, gateway address and the round trip time of each of the default three probes; a probe with no response in time prints an asterisk; -m caps the hops, -q sets probes per hop, -w caps the wait, and some routers use ICMP rate throttling.
- 01The TTL trick
- 02What stars mean
Per Linux man-pages traceroute(8): traceroute tracks the route packets take to a host by utilizing the IP protocol's time to live field and attempting to elicit an ICMP TIME_EXCEEDED response from each gateway along the path; it launches probe packets with a small ttl, starts at one and increases by one until an ICMP port unreachable or TCP reset means the host is reached, or a max that defaults to 30 hops. Confirmed on traceroute 2.1.5.
The TTL trick
traceroute repurposes a safety limit as a measuring stick. A probe sent with ttl=1 dies at the first router, which must say so: the ICMP time exceeded reply names it. ttl=2 dies one hop further, and so on.
When the probe finally reaches the host, the reply changes to port unreachable, and the walk stops, at 30 hops by default.
Nobody volunteers a hop list; traceroute makes each router declare itself as it discards your packet.
Run traceroute -n -q 1 -w 2 1.1.1.1, count the hops, and point at the line that proves the first router answered a ttl=1 probe.
Per Linux man-pages traceroute(8): if there is no response from a probe within a certain timeout, an asterisk is printed for that probe. On this machine's traceroute 2.1.5 run to 1.1.1.1 with udp probes, hop 5 printed only stars while hops 4 and 6 answered normally.
What stars mean
A star is one probe that got no answer in time. That is all it means: no reply, not no router, and not no path.
Our run is the proof: hop 5 printed *, and hop 6 answered immediately after. Traffic flowed through the silent hop while it declined to speak.
A star mid-path is a router ignoring strangers; stars all the way to the end, with no final hop, is a different and real problem.
Run traceroute -n -q 1 -w 2 1.1.1.1, find the star hop, and confirm the hops after it still answer before you suspect the path.
Per Linux man-pages traceroute(8): -T, --tcp uses TCP SYN for probes, and reaching the host is signaled by a TCP reset or the configured port answering; methods use a particular protocol and ports in order to be seen by filters as an allowed type of session. On this machine, the default udp run to 1.1.1.1 drew a star at hop 5 while traceroute -T named all 6 hops on traceroute 2.1.5.
TCP probes (-T)
-T sends TCP SYN probes, the opening handshake of a connection, so filters that ignore udp or icmp see what looks like ordinary traffic.
Our pair of runs: udp drew a star at hop 5; tcp named hop 5 at 2.837 ms and hop 6 at 2.727 ms. Same destination, same path, different dialect, full answer.
Before deciding a hop is invisible, ask it in all three dialects: udp, icmp, tcp.
Run traceroute -n -q 1 1.1.1.1 and then traceroute -T -n -q 1 1.1.1.1, and count the hops each dialect managed to name.
Per mtr 0.95 man page: a sudden increase in packet loss or response time is often an indication of a bad or simply overloaded link. Per Linux man-pages traceroute(8): some routers and hosts use ICMP rate throttling, and too many probes can lose responses. On this machine's mtr -r -n -c 10 report to 1.1.1.1, hop 1 shows Loss% 10.0 with 0.0% at every hop after it.
Loss per hop
The Loss% column counts missing replies per hop. The man page warns that a sudden increase in loss or response time points to a bad or overloaded link.
But our report shows the other case: 10.0% at hop 1, 0.0% at every hop after. Later hops saw all ten probes, so the packets got through; only hop 1's replies were withheld, the rate throttling traceroute(8) documents.
Real path loss continues to the end of the report; loss that stops mid-column is a quiet router, not a broken link.
Run mtr -r -n -c 10 1.1.1.1 and read the Loss% column top to bottom; decide whether any loss you see continues or stops.
Per Linux man-pages ss(8): ss is used to dump socket statistics; -t selects TCP sockets; -l shows only listening sockets; -n does not try to resolve service names; -p shows the processes using the sockets. This machine's run on iproute2 6.1.0 lists sshd listening on 0.0.0.0:22, apache2 on :80 and :443, and redis-server on 127.0.0.1:6379.
Who listens (-tlnp)
Four letters, one inventory. -t narrows to TCP, -l to listeners, -n keeps ports numeric, and -p names the process behind each socket.
Our run answers who is open for business: sshd on :22 for the world, apache2 on :80 and :443, and redis-server bound to 127.0.0.1 only, closed to outsiders.
Before asking why a connection fails, know what was listening, on which address, in which process.
Run ss -tlnp on a machine you own, find your sshd line, and note which listeners are bound to 127.0.0.1 rather than 0.0.0.0.
Per Linux man-pages ip-route(8): there is a special PREFIX default, which is equivalent to IP 0/0 or IPv6 ::/0; routes name a gateway with via and a device with dev. This machine's table, on iproute2 6.1.0, reads default via 194.195.90.1 dev eth0 proto static.
The default route
The routing table's last resort: default via 194.195.90.1 dev eth0. Per ip-route(8), default is the prefix 0/0, matching every address there is.
The line names the door: hand the packet to 194.195.90.1 through eth0. Anything no more specific route claims leaves this way.
No default route and every far address is unreachable at once; check it first when nothing outside works.
Run ip route show, find your default line, and write down the gateway address your unmatched traffic leaves through.
This book's plan for a slow page: ping(8) checks the name and the round trip per its documented fault isolation order; the mtr 0.95 man page's Loss% and Wrst columns locate the hop that hurts; ss(8) shows whether the local end has stuck queues. Three tools, each used exactly as its man page documents, in order.
The slow page run
A page is slow. Resist the packet-capture reflex; walk the ladder. 1. ping the hostname: did it resolve, what RTT? 2. mtr -r -n -c 10 to it: which hop's Wrst or Loss% stands out?
3. ss -tn: are the local queues clear? Each answer eliminates a whole class of causes.
DNS slow makes every page slow; loss late in the path names one hop; full queues are the application's own traffic. The order is the diagnosis.
Next time a page drags, run ping -c 2 to its host, then mtr -r -n -c 10, and write down which single column explained the wait.