← All 163 books wget, one download per page
One download per page

wget, one download per page

Twenty-eight lessons on what wget is: a non-interactive retriever over HTTP, HTTPS and FTP; Geturl in 1996 and why an unattended downloader earns its keep; the law of fetching only what you own or may fetch; one URL, one command: the plain fetch, a chosen name with -O, a file of URLs with -i, the -o log; what breaks, the Range idea, the -c resume and one honest file verified; the polite guest: -U, --limit-rate, the -w -t -T dials, robots.txt; the mirror idea: -r, -l, -p, -k and -m; quiet and verbose, exit statuses, the 404 answer and curl named honestly; three protocols; an offline copy; one keeper command; teaching a friend in one sitting.


Steve Hodgkiss 7 lessons

One URL you may fetch, one honest hour. That's the rhythm.

wget, one download per page

Twenty-eight lessons on what wget is: a non-interactive retriever over HTTP, HTTPS and FTP; Geturl in 1996 and why an unattended downloader earns its keep; the law of fetching only what you own or may fetch; one URL, one command: the plain fetch, a chosen name with -O, a file of URLs with -i, the -o log; what breaks, the Range idea, the -c resume and one honest file verified; the polite guest: -U, --limit-rate, the -w -t -T dials, robots.txt; the mirror idea: -r, -l, -p, -k and -m; quiet and verbose, exit statuses, the 404 answer and curl named honestly; three protocols; an offline copy; one keeper command; teaching a friend in one sitting.


Set in Space Grotesk, Inter and JetBrains Mono (SIL Open Font License).

Every fact is as the official source states it, fetched and read during this build: the local wget --help and wget man page of GNU Wget 1.21.4 on this machine, the truth for flags, exit statuses and the robots law, and the Wikipedia articles Wget, HTTP, HTTPS, FTP, Byte serving, HTTP 404, cURL, Mirror site, Robots.txt, Web crawler, GNU Project and Web archiving. No facts come from Reddit, Stack Overflow, vendor blogs or commercial sites. Every download quoted ran against a local http.server bound to this machine's loopback only (127.0.0.1:8471, Range support added so -c could truly resume) and is cached verbatim beside this book.

Honest notes: no HTTPS and no FTP download ran, the demo server spoke plain HTTP on loopback; no robots.txt file existed, and the captures show wget asking for it and continuing on the 404; no -o log and no -w wait demo ran, both stated on their pages. Disclosed redirect and disambiguation stubs were not used as fact sources. SAFETY LAW: no third-party host was ever contacted. Independent, not affiliated with the GNU Project or the Wikimedia Foundation.

Your purchase is for personal use only. You do not have redistribution rights: please do not share, resell, or republish this book or its pages.

© 2026 Steve Hodgkiss. All rights reserved. Personal use only; no redistribution rights.

Edition 1.0 · stevehodgkiss.net

Contents

Contents


Part 1 · The honest tool
The law first4
Part 2 · One URL, one command
A list of URLs5
Part 3 · The broken download
One honest file6
Part 4 · The polite guest
robots.txt7
Part 5 · The mirror idea
The -m shortcut8
Part 6 · Reading honestly
Exit statuses9
Part 7 · The long view
Teaching one download10

Per the Wikipedia Robots.txt article, robots.txt is a standard used by websites to indicate to visiting crawlers and web robots which portions of the website they are allowed to visit. Per the Wikipedia Web crawler article, crawler problems include server overload, especially if the frequency of accesses to a given server is too high. Per this book's plan, the law is: fetch only what you own or are allowed to fetch, and ask before mirroring anything else.

Wget companion · No. 01
the law

The law first

fetch only what you may

the law before the first byte: whose server is ithammering: server overloadwhen accesses come too fastthis book's law: fetch onlywhat you own or may fetchsites publish their own rules:robots.txt says which portionsvisitors are allowed to visitthe owner's answer settles it;if not you and not a yes,the download does not runevery capture in this book fetched from this machine's own loopback server

Before the first fetch, the law. Sites publish their own rules: robots.txt tells crawlers which portions of the website they are allowed to visit.

The red edge is real: crawlers cause server overload, especially if the frequency of accesses to a given server is too high. A downloader can hammer a machine that never agreed.

So this book's law, per this book's plan: fetch only what you own or are allowed to fetch. Every capture here fetched from this machine's own loopback server.

ONE HONEST DOWNLOAD

Before your next download, say aloud whose server it is; if the answer is not you or a yes from the owner, stop.

Per the local wget --help, -i or --input-file=FILE downloads URLs found in local or external FILE. This machine's capture: a two-line urls.txt holding two loopback URLs, run with wget -i, saved hello.txt at 85 bytes and one.txt at 9 bytes, finishing with Downloaded: 2 files, exit status 0.

Wget companion · No. 02
the batch

A list of URLs

-i reads a file of URLs

a list is a plan: -i reads the URLs from a fileurls.txthello.txt one.txttwo loopback URLs,one per line-ihello.txt 85one.txt 9Downloaded: 2 files (captured)the help text:-i, --input-file=FILEdownload URLs found inlocal or external FILE;write the plan, run oncemany URLs, still one command: the list is the plan

Many URLs, still one command. Per the local help text: -i, --input-file=FILE will download URLs found in local or external FILE, one per line.

Our capture: wget -i urls.txt read two loopback URLs and saved both: hello.txt saved [85/85] and one.txt saved [9/9], finishing Downloaded: 2 files.

The list is a plan you can reread: write URLs down first, then let the one command do the fetching.

ONE HONEST DOWNLOAD

Write tomorrow's three downloads into one urls.txt tonight; fetch them with one command in the morning.

Per this book's plan, a download is not done until the file on disk is verified. This machine's checks: the resumed 131072-byte big.bin hashed identically to the server's copy, sha256 59f410ae5e17962412e2aed4f815918f634932f2abf084f00bb638c4db017850 on both sides, recorded in the local checks file. Per the local wget --help, -S or --server-response prints the server response, one way to see what the server claimed it sent.

Wget companion · No. 03
the check

One honest file

verify what landed

one honest file: hash both sides, then trustserverbig.bin on the serverbig.binthe resumed copysha256, both sides:59f410ae5e17962412e2aed4f815918f6 (captured,both full hashes equal)the habit, per thisbook's plan: size,then hash, then trust;-S shows the server'sown claim about whatit sent; measure anywayfetched bytes are a stranger until measured

Teal is the outcome: one honest file, verified. Our check, per the local checks file: the server's copy and the resumed copy both hashed 59f410ae5e17962412e2aed4f815918f6, the same full sha256 on both sides. Identical, byte for byte.

The tool helps you look: per the local help text, -S, --server-response will print the server response, the server's own claim about what it sent.

Per this book's plan: size, then hash, then trust. Fetched bytes are a stranger until measured.

ONE HONEST DOWNLOAD

Hash one file you downloaded this week and write the first eight hex digits beside its log line.

Per the local wget man page, while recursing, Wget respects the Robot Exclusion Standard, the robots.txt file. Per the Wikipedia Wget article, Wget supports the Robots Exclusion Standard unless the option -e robots=off is used; this book never types it. Per the Wikipedia Robots.txt article, the standard was proposed by Martijn Koster, developed in 1994, and relies on voluntary compliance. This machine's captures: both recursive runs asked for /robots.txt first and the demo server, having none, answered 404.

Wget companion · No. 04
the guest law

robots.txt

the site's own rules

the house rules first: robots.txt, asked before any pagewget/robots.txtasked first, alwaysthe pagesif allowedasked: http://127.0.0.1:8471/robots.txtanswer: ERROR 404 (no rules on this server)wget continued: no rules means nothing forbiddenthe 1994 standard,proposed by MartijnKoster, relies onvoluntary compliance;this book never typesrobots=offthe guest asks the house rules before entering; comply voluntarily

The guest asks the house rules first. Per the local man page, while recursing, Wget respects the Robot Exclusion Standard (/robots.txt); per the article, unless the option -e robots=off is used, which this book never types.

Our captures: both recursive runs asked for http://127.0.0.1:8471/robots.txt before any page; the demo server had none, answered ERROR 404, and wget continued.

The standard relies on voluntary compliance, proposed by Martijn Koster in 1994. Comply voluntarily.

ONE HONEST DOWNLOAD

Fetch any site's robots.txt by hand before your next recursive run and read its rules aloud.

Per the local wget --help, -m or --mirror is a shortcut for -N -r -l inf --no-remove-listing, and -N or --timestamping does not re-retrieve files unless newer than local. Per the Wikipedia Wget article, recursive download enables partial or complete mirroring of web sites via HTTP, and Wget can inspect the timestamps of local and remote files and download only the remote files newer than the corresponding local ones. Per the Wikipedia Mirror site article, mirror sites are replicas of other websites that host identical or near-identical content. Per this book's plan, mirror only what you own.

Wget companion · No. 05
the mirror

The -m shortcut

-m mirrors, carefully

the -m shortcut: one flag that means four habits-m-N-r-l inf--no-remove-listingnewer than local onlyrecurse the siteno depth floorkeep the listingsred line, this book's plan:mirror only what you ownmirrors are replicas of othersites; recursion enables partialor complete mirroring via HTTPknow what the shortcut expands to before you type it

The mirror habit in one flag. Per the local help text, -m, --mirror is a shortcut for -N -r -l inf --no-remove-listing: recurse without a floor, and with -N, don't re-retrieve files unless newer than local.

Per the article, recursion enables partial or complete mirroring of web sites via HTTP, taking only what is newer than the corresponding local ones.

Mirrors are replicas of other websites. Red line, per this book's plan: mirror only what you own.

ONE HONEST DOWNLOAD

List what -m expands to from memory; then mirror one directory of your own site as practice.

Per the local wget man page, Wget may return one of several error codes: 0 no problems occurred, 1 generic error, 4 network failure, 8 server issued an error response, and with the exceptions of 0 and 1 the lower-numbered codes take precedence. This machine's captures: the plain fetch exited 0, the 404 fetch exited 8, and the refused connection exited 4.

Wget companion · No. 06
the codes

Exit statuses

0 through 8, honestly

the exit codes, captured: 0 saved, 4 refused, 8 a server error0no problems occurred‘hello.txt’ saved [85/85]4network failurefailed: Connection refused.8server issued an error responseERROR 404: File not found.1 generic; 2 parse; 3 file I/O; 5 SSL; 6 auth; 7 protocolread the number:it separates yournetwork (4) fromtheir server (8);lower numbers takeprecedence over higha script that checks nothing has learned nothing

The command answers with one number. Per the man page, 0 No problems occurred; 4 Network failure; 8 Server issued an error response; and the lower-numbered exit codes take precedence over higher-numbered ones.

Our captures: the plain fetch exited 0; the refused connection exited 4; the missing page exited 8.

A script that checks nothing has learned nothing. Read the number; it separates your network from their server.

ONE HONEST DOWNLOAD

Write a two-line habit that echoes the exit status after every fetch you script this week.

Per this book's plan, the last lesson hands wget to a friend in one honest hour: five captured lines from this book, typed by the learner against a server the learner owns, read aloud by the teacher. The captures behind the five lines: the plain fetch's saved line, the -O rename, the 206 Partial Content resume, the 404 error with exit 8, and the refused connection with exit 4.

Wget companion · No. 07
pass it on

Teaching one download

one honest hour, one friend

the last hour: five captured lines, one friend, taught1‘hello.txt’ saved [85/85]2-O greeting.txt: the chosen name3206 Partial Content: the resume4ERROR 404, exit 8: the server spoke5Connection refused, exit 4: it did notthey can say why 4 and 8 differthey type, on a serverthey own; you readeach answer aloud;one honest hour,per this book'splan, and thetool is theirswhen they can say why the guess was wrong, the hour is done

The last piece is passing the tool on: one friend, one honest hour, per this book's plan. They type; you read aloud.

Five lines, this book's captures: the plain fetch ending hello.txt saved [85/85], the -O rename to greeting.txt, the resume's 206 Partial Content, the ERROR 404 with exit 8, and the failed: Connection refused. with exit 4.

When they can say why 4 and 8 differ, the hour is done and the downloader is theirs.

ONE HONEST DOWNLOAD

Sit with one friend and teach this book in one sitting; they type the five lines, you read each answer aloud.

Index

Index


A list of URLs5
Exit statuses9
One honest file6
robots.txt7
Teaching one download10
The -m shortcut8
The law first4