8. Final project: remote diagnostics toolkit

Description

In this lesson you are going to build ops-toolkit: three shell scripts that combine to diagnose whether a remote server is healthy, and a README.md that documents how to use it and how to defend it in front of someone else. check-env.sh audits that a machine has what it needs to operate (dependencies, PATH, environment variables, permissions on a private key). probe.sh diagnoses a domain layer by layer (DNS, HTTP, local port). remote-report.sh connects both over SSH to a remote machine, brings back the results with rsync, and builds a summary with grep, sort, and uniq -c. Each script exits with a different code depending on what failed, documented in a table — not "it went wrong," but "it went wrong because a dependency is missing" versus "it went wrong because DNS does not resolve."

This is exactly what an engineer does when the message "the site is not responding" arrives at 11 at night: they do not open an IDE, they open a terminal, connect to the server, and run a handful of commands in an order they already know by heart because they automated it a while ago. The difference between "it took me forty minutes typing the usual stuff" and "I ran a script and in ten seconds I knew it was DNS" is exactly the content of this module.

Connection to the module: this project does not teach anything new — it wires together everything you already saw. Environment variables and file permissions (lesson 1), IP/ports/DNS (lesson 2), dig/curl/ss (lesson 3), SSH with keys and ~/.ssh/config (lesson 4), rsync (lesson 5), and arguments, case, loops, exit codes, set -euo pipefail, and trap (lessons 6 and 7) turn here into a single tool that runs from end to end.

The toolkit's architecture: three layers, one responsibility each

Think of an electrician called to a house they have never seen. They do not start by tearing open walls. First they check the electrical panel: is power coming in, are the breakers where they should be, is the panel labeled or is it chaos? That is check-env.sh — it does not diagnose anything about the actual problem, it just confirms the tools and the environment are in shape for a real diagnosis. Then they test each outlet separately, in order: does power reach the box (DNS), does the appliance respond when you plug it in (HTTP), is the specific breaker switched on (local port)? That is probe.sh. And if the house is in another city, the electrician does not drive out there empty-handed: they send someone with the same tools, that person checks on site, and the result comes back by courier. That is remote-report.sh.

The reason there are three scripts and not one with five hundred lines is the same one you saw in lesson 7 about when a script has "already grown too big": each one does one thing, each one can be tested on its own, and remote-report.sh does not repeat the diagnostic logic — it reuses it by copying it to the remote machine and running it there. It is the Unix philosophy applied to your own tools, not just to other people's.

One warning before you start: ss is part of iproute2, exclusive to Linux. If you write and test these scripts from macOS, layer 3 of probe.sh is going to fail locally with "command not found" — which is exactly why check-env.sh verifies its presence instead of assuming it. The real design is that you run probe.sh on the server (almost always Linux) through remote-report.sh, and use it on your laptop only for the "from outside" view (DNS and HTTP).

Create a folder ops-toolkit/ and build this inside it as you go:

ops-toolkit/
├── check-env.sh
├── probe.sh
├── remote-report.sh
├── domains.txt
└── README.md

Worked example: check-env.sh, the entry gate

check-env.sh does not test the network or servers — it tests that your own machine is ready to operate the toolkit. It checks four things, in order: that PATH includes the basics, that the commands the rest of the toolkit needs actually exist, that the TOOLKIT_SSH_KEY variable (the path to your private key) is defined, and that key has permissions only you can read.

#!/usr/bin/env bash
# check-env.sh — audits that this machine is ready to run ops-toolkit:
# PATH, required commands, environment variables, and SSH key permissions.
set -euo pipefail

# Exit codes for the whole toolkit (also documented in README.md).
readonly EX_OK=0
readonly EX_USAGE=64       # invalid arguments
readonly EX_UNAVAILABLE=69 # a required command is missing
readonly EX_NOPERM=77      # unsafe permissions on a sensitive file
readonly EX_CONFIG=78      # an environment variable is missing

SCRIPT_NAME="$(basename "$0")"
DRY_RUN=0

usage() {
  cat <<EOF
Usage: ${SCRIPT_NAME} [--dry-run] [--help]

Checks that this machine has what it needs to run ops-toolkit:
required commands on PATH, the TOOLKIT_SSH_KEY variable, and safe
permissions on the private key it points to.

  --dry-run   Shows what would be checked, without aborting on any failure.
  --help      Shows this help and exits with code 0.

Exit codes: see the table in README.md.
EOF
}

while [[ $# -gt 0 ]]; do
  case "$1" in
    --dry-run) DRY_RUN=1; shift ;;
    --help) usage; exit "${EX_OK}" ;;
    *)
      echo "Error: unknown argument '$1'" >&2
      usage >&2
      exit "${EX_USAGE}"
      ;;
  esac
done

fail() {
  local code="$1" msg="$2"
  echo "FAILED: ${msg}" >&2
  if [[ "${DRY_RUN}" -eq 1 ]]; then
    echo "  (dry-run: not aborting, but in normal mode it would exit with code ${code})"
  else
    exit "${code}"
  fi
}

echo "== PATH =="
echo "  current PATH: ${PATH}"
for dir in /usr/bin /bin; do
  case ":${PATH}:" in
    *":${dir}:"*) echo "  ok: ${dir} is in PATH" ;;
    *) echo "  warning: ${dir} is not in PATH (typical if this runs under cron)" ;;
  esac
done

echo "== dependencies =="
REQUIRED_COMMANDS=(dig curl ss ssh rsync)
for cmd in "${REQUIRED_COMMANDS[@]}"; do
  if command -v "${cmd}" >/dev/null 2>&1; then
    echo "  ok: ${cmd} -> $(command -v "${cmd}")"
  else
    fail "${EX_UNAVAILABLE}" "missing command '${cmd}' on PATH"
  fi
done

echo "== TOOLKIT_SSH_KEY environment variable =="
if [[ -z "${TOOLKIT_SSH_KEY:-}" ]]; then
  fail "${EX_CONFIG}" "TOOLKIT_SSH_KEY is not set (example: export TOOLKIT_SSH_KEY=\$HOME/.ssh/id_ed25519)"
else
  echo "  ok: TOOLKIT_SSH_KEY=${TOOLKIT_SSH_KEY}"
fi

echo "== private key permissions =="
if [[ -n "${TOOLKIT_SSH_KEY:-}" && -f "${TOOLKIT_SSH_KEY}" ]]; then
  # %Lp (BSD/macOS) or %a (GNU/Linux) return the octal mode, e.g. "600".
  # Assumes the key has no setuid/sticky bit, which is the normal case.
  PERMS="$(stat -f "%Lp" "${TOOLKIT_SSH_KEY}" 2>/dev/null || stat -c "%a" "${TOOLKIT_SSH_KEY}")"
  if [[ "${PERMS}" =~ ^[0-7]00$ ]]; then
    echo "  ok: permissions ${PERMS} (only the owner can read/write)"
  else
    fail "${EX_NOPERM}" "${TOOLKIT_SSH_KEY} has permissions ${PERMS}; fix with: chmod 600 ${TOOLKIT_SSH_KEY}"
  fi
elif [[ -n "${TOOLKIT_SSH_KEY:-}" ]]; then
  fail "${EX_CONFIG}" "TOOLKIT_SSH_KEY points to '${TOOLKIT_SSH_KEY}', which does not exist"
fi

echo "check-env.sh: everything is fine."
exit "${EX_OK}"

How to read it: the fail() function centralizes the message ("FAILED: ...") and the exit code in one single place, and it respects --dry-run — instead of deciding at every failure point whether to abort or not, every call to fail() already knows what to do based on the mode. The permissions check uses a regular expression (^[0-7]00$) instead of comparing numbers, because comparing "640" against "600" with -gt compares decimal integers, not permission bits — it works most of the time by coincidence, but the regex says exactly what you mean: "the last two digits must be zero."

What to expect when you run it with the variable set correctly and the key at 600:

$ chmod +x check-env.sh
$ export TOOLKIT_SSH_KEY=$HOME/.ssh/id_ed25519
$ ./check-env.sh
== PATH ==
  current PATH: /usr/local/bin:/usr/bin:/bin
  ok: /usr/bin is in PATH
  ok: /bin is in PATH
== dependencies ==
  ok: dig -> /usr/bin/dig
  ok: curl -> /usr/bin/curl
  ok: ss -> /usr/sbin/ss
  ok: ssh -> /usr/bin/ssh
  ok: rsync -> /usr/bin/rsync
== TOOLKIT_SSH_KEY environment variable ==
  ok: TOOLKIT_SSH_KEY=/home/ana/.ssh/id_ed25519
== private key permissions ==
  ok: permissions 600 (only the owner can read/write)
check-env.sh: everything is fine.
$ echo $?
0

And if the key ended up with 644 permissions (readable by anyone in the group):

$ chmod 644 ~/.ssh/id_ed25519
$ ./check-env.sh; echo "exit: $?"
...
== private key permissions ==
FAILED: /home/ana/.ssh/id_ed25519 has permissions 644; fix with: chmod 600 /home/ana/.ssh/id_ed25519
exit: 77

The 77 is not arbitrary — it is the same code you are going to document in the README.md table, and it is the one remote-report.sh is going to interpret later on.

probe.sh: diagnosing a domain layer by layer

The idea of "diagnosing layer by layer" comes straight from lesson 3: when something does not respond, the order of suspicion goes from the most basic to the most specific. If the name does not resolve, there is no point testing HTTP. If HTTP responds, there is no point suspecting DNS. probe.sh automates exactly that elimination order, splitting it into three layers: dig for DNS, curl -w for HTTP with timing, and ss to confirm whether a process is listening locally on the port — this last one meant for when you run the script on the server itself (via remote-report.sh), where it tells you whether the problem is that the service never started, rather than a network issue.

#!/usr/bin/env bash
# probe.sh — diagnoses a domain across three layers: DNS, HTTP, and the local
# port's state. Results go to stdout, errors to stderr.
set -euo pipefail

readonly EX_OK=0
readonly EX_USAGE=64
readonly EX_UNAVAILABLE=69 # DNS does not resolve
readonly EX_SOFTWARE=70    # the HTTP request failed (timeout, connection refused)

SCRIPT_NAME="$(basename "$0")"
PORT=""
SCHEME="https"
DRY_RUN=0
DOMAIN=""

usage() {
  cat <<EOF
Usage: ${SCRIPT_NAME} <domain> [--port PORT] [--scheme http|https] [--dry-run] [--help]

Diagnoses a domain across three layers: DNS (dig), HTTP (curl -w), and the
local port's state (ss).

  --port PORT         Port for ss (default: 443 if scheme=https, 80 if http)
  --scheme http|https Scheme for the curl request (default: https)
  --dry-run           Shows the commands, without running them
  --help               Shows this help and exits with code 0

Exit codes: see the table in README.md.
EOF
}

fail() {
  local code="$1" msg="$2"
  echo "FAILED: ${msg}" >&2
  exit "${code}"
}

while [[ $# -gt 0 ]]; do
  case "$1" in
    --port) PORT="$2"; shift 2 ;;
    --scheme) SCHEME="$2"; shift 2 ;;
    --dry-run) DRY_RUN=1; shift ;;
    --help) usage; exit "${EX_OK}" ;;
    -*) fail "${EX_USAGE}" "unknown option '$1'" ;;
    *)
      [[ -n "${DOMAIN}" ]] && fail "${EX_USAGE}" "only one domain is accepted, already got '${DOMAIN}'"
      DOMAIN="$1"
      shift
      ;;
  esac
done

[[ -z "${DOMAIN}" ]] && { usage >&2; fail "${EX_USAGE}" "missing the domain to diagnose"; }
[[ -z "${PORT}" ]] && { [[ "${SCHEME}" == "https" ]] && PORT=443 || PORT=80; }

echo "== Layer 1/3: DNS (${DOMAIN}) =="
if [[ "${DRY_RUN}" -eq 1 ]]; then
  echo "[dry-run] dig +short ${DOMAIN}"
else
  # dig can fail (network down) or simply have no records; in both cases IP
  # ends up empty and we treat it as a single failure condition.
  IP="$(dig +short "${DOMAIN}" | tail -n1 || true)"
  [[ -z "${IP}" ]] && fail "${EX_UNAVAILABLE}" "no record resolves for '${DOMAIN}'"
  echo "resolves to: ${IP}"
fi

echo "== Layer 2/3: HTTP (${SCHEME}://${DOMAIN}) =="
if [[ "${DRY_RUN}" -eq 1 ]]; then
  echo "[dry-run] curl -o /dev/null -s -w '...' --max-time 5 ${SCHEME}://${DOMAIN}"
else
  if ! curl -o /dev/null -s --max-time 5 \
      -w 'http_code=%{http_code} time_namelookup=%{time_namelookup}s time_connect=%{time_connect}s time_total=%{time_total}s\n' \
      "${SCHEME}://${DOMAIN}"; then
    fail "${EX_SOFTWARE}" "curl could not complete the request to ${SCHEME}://${DOMAIN} (timeout or connection refused)"
  fi
fi

echo "== Layer 3/3: local port ${PORT} (ss) =="
if [[ "${DRY_RUN}" -eq 1 ]]; then
  echo "[dry-run] ss -tuln | grep -E ':${PORT}\b'"
else
  if ss -tuln | grep -qE ":${PORT}\b"; then
    echo "there is a process listening locally on port ${PORT}"
  else
    echo "nothing is listening locally on port ${PORT} (normal if this runs outside the server)"
  fi
fi

echo "OK"
exit "${EX_OK}"

A detail worth looking at twice: I use tail -n1, not head -n1, on dig +short's output. When a domain has a CNAME chain, dig +short prints the name it points to first and then the final IP — taking the first line gets you a hostname, not an address. tail -n1 assumes the chain ends in an A record, which is the normal case, but it is not foolproof (we come back to this in common mistakes).

What to expect against a real domain:

$ ./probe.sh example.com
== Layer 1/3: DNS (example.com) ==
resolves to: 93.184.216.34
== Layer 2/3: HTTP (https://example.com) ==
http_code=200 time_namelookup=0.012s time_connect=0.045s time_total=0.187s
== Layer 3/3: local port 443 (ss) ==
nothing is listening locally on port 443 (normal if this runs outside the server)
OK
$ echo $?
0

And against a domain that does not exist:

$ ./probe.sh domain-that-does-not-exist-xyz.test
== Layer 1/3: DNS (domain-that-does-not-exist-xyz.test) ==
FAILED: no record resolves for 'domain-that-does-not-exist-xyz.test'
$ echo $?
69

remote-report.sh: orchestrating the remote diagnosis

This script does not diagnose anything by itself — it orchestrates. It takes an alias from ~/.ssh/config and a file with a list of domains (one per line), and for each one: runs check-env.sh once on the remote as an entry gate, runs probe.sh for each domain saving the output to files on the server itself, brings everything back with a single rsync, and builds a local summary with grep | sort | uniq -c. The reason for writing the results to disk on the remote and bringing them back afterward with rsync — instead of letting ssh stream the output straight to your terminal — is deliberate: that way the "retrieve results" exercise is real, not cosmetic, and if the connection drops halfway through a long diagnostic run, the results of whatever already ran are still there to recover afterward.

#!/usr/bin/env bash
# remote-report.sh — runs check-env.sh and probe.sh on a remote server for a
# list of domains, brings back the results with rsync, and summarizes with grep/sort/uniq.
set -euo pipefail

readonly EX_OK=0
readonly EX_USAGE=64
readonly EX_UNAVAILABLE=69 # check-env.sh failed on the remote
readonly EX_CONFIG=78      # invalid SSH alias or domains file

SCRIPT_NAME="$(basename "$0")"
# Resolves the folder this script lives in, no matter where you invoke it from.
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
DRY_RUN=0

usage() {
  cat <<EOF
Usage: ${SCRIPT_NAME} <ssh-alias> <domains-file> [--dry-run] [--help]

Connects to <ssh-alias> (must exist as a 'Host' in ~/.ssh/config, lesson 4),
runs check-env.sh and probe.sh for each domain in <domains-file> (one per
line), brings back the results with rsync, and shows a final summary.

  --dry-run   Shows what would be run, without connecting to anything.
  --help      Shows this help and exits with code 0.

Exit codes: see the table in README.md.
EOF
}

fail() {
  local code="$1" msg="$2"
  echo "FAILED: ${msg}" >&2
  exit "${code}"
}

POSITIONAL=()
while [[ $# -gt 0 ]]; do
  case "$1" in
    --dry-run) DRY_RUN=1; shift ;;
    --help) usage; exit "${EX_OK}" ;;
    -*) fail "${EX_USAGE}" "unknown option '$1'" ;;
    *) POSITIONAL+=("$1"); shift ;;
  esac
done
set -- "${POSITIONAL[@]}"
[[ $# -ne 2 ]] && { usage >&2; fail "${EX_USAGE}" "exactly 2 arguments are expected: alias and domains file"; }
ALIAS="$1"
DOMAINS_FILE="$2"

[[ -f "${DOMAINS_FILE}" ]] || fail "${EX_CONFIG}" "domains file '${DOMAINS_FILE}' does not exist"
# Validated even under --dry-run: it is a local read, costs no network trip,
# and catches the most common configuration mistake before touching anything.
grep -qE "^Host[[:space:]]+${ALIAS}([[:space:]]|\$)" "${HOME}/.ssh/config" 2>/dev/null \
  || fail "${EX_CONFIG}" "'${ALIAS}' is not defined as a Host in ~/.ssh/config (lesson 4)"

if [[ "${DRY_RUN}" -eq 1 ]]; then
  echo "[dry-run] ssh ${ALIAS} mktemp -d"
  echo "[dry-run] rsync check-env.sh probe.sh -> ${ALIAS}:<remote>/"
  echo "[dry-run] ssh ${ALIAS} bash check-env.sh"
  while IFS= read -r domain || [[ -n "${domain}" ]]; do
    [[ -z "${domain}" || "${domain}" == \#* ]] && continue
    echo "[dry-run] ssh ${ALIAS} bash probe.sh ${domain}"
  done < "${DOMAINS_FILE}"
  echo "[dry-run] rsync ${ALIAS}:<remote>/results/ -> ./reports/<timestamp>/"
  exit "${EX_OK}"
fi

TIMESTAMP="$(date +%Y%m%d-%H%M%S)"
LOCAL_REPORTS_DIR="./reports/${TIMESTAMP}"
REMOTE_DIR=""

# Unlike check-env.sh and probe.sh, this trap does not delete the final
# report — it only cleans up what this script left on the remote server.
cleanup() {
  [[ -n "${REMOTE_DIR}" ]] && ssh "${ALIAS}" "rm -rf '${REMOTE_DIR}'" 2>/dev/null || true
}
trap cleanup EXIT

mkdir -p "${LOCAL_REPORTS_DIR}"

echo "== Preparing remote directory on ${ALIAS} =="
REMOTE_DIR="$(ssh "${ALIAS}" mktemp -d)"
ssh "${ALIAS}" "mkdir -p '${REMOTE_DIR}/results'"

echo "== Copying scripts with rsync =="
rsync -az "${SCRIPT_DIR}/check-env.sh" "${SCRIPT_DIR}/probe.sh" "${ALIAS}:${REMOTE_DIR}/"

echo "== Running check-env.sh on the remote =="
if ! ssh "${ALIAS}" "bash '${REMOTE_DIR}/check-env.sh'" \
    > "${LOCAL_REPORTS_DIR}/check-env.out" 2> "${LOCAL_REPORTS_DIR}/check-env.err"; then
  cat "${LOCAL_REPORTS_DIR}/check-env.err" >&2
  fail "${EX_UNAVAILABLE}" "check-env.sh failed on '${ALIAS}'; check ${LOCAL_REPORTS_DIR}/check-env.err"
fi

echo "== Diagnosing domains =="
while IFS= read -r domain || [[ -n "${domain}" ]]; do
  [[ -z "${domain}" || "${domain}" == \#* ]] && continue
  echo "  -- ${domain} --"
  # The '||' is intentional: if one domain fails, the rest of the list must
  # keep running. Without it, set -e would take down the whole batch on the first failure.
  ssh "${ALIAS}" \
    "bash '${REMOTE_DIR}/probe.sh' '${domain}' > '${REMOTE_DIR}/results/${domain}.out' 2> '${REMOTE_DIR}/results/${domain}.err'" \
    || echo "     (probe.sh reported a failure for ${domain}; continuing with the rest)"
done < "${DOMAINS_FILE}"

echo "== Bringing back results with rsync =="
rsync -az "${ALIAS}:${REMOTE_DIR}/results/" "${LOCAL_REPORTS_DIR}/"

echo "== Summary =="
echo "-- Failures recorded --"
grep -h "^FAILED" "${LOCAL_REPORTS_DIR}"/*.err 2>/dev/null | sort | uniq -c | sort -rn || echo "  (none)"
echo "-- HTTP codes seen --"
grep -hoE 'http_code=[0-9]+' "${LOCAL_REPORTS_DIR}"/*.out 2>/dev/null | sort | uniq -c || echo "  (none)"

echo "Full results in: ${LOCAL_REPORTS_DIR}"
exit "${EX_OK}"

The point most worth defending out loud if someone asks you about this script: the || at the end of the ssh line inside the loop is not decoration. Under set -e, a command that fails inside a loop without that || ends the entire script the moment the first domain in the list fails — the rest of the list never gets tested, and the report looks like "the script broke" when in reality only one domain was down. The || turns that individual failure into a data point in the report, not an accident.

What to expect with a three-line domains.txt where one is down:

$ cat domains.txt
example.com
example.org
domain-that-does-not-exist-xyz.test

$ ./remote-report.sh webserver domains.txt
== Preparing remote directory on webserver ==
== Copying scripts with rsync ==
== Running check-env.sh on the remote ==
== Diagnosing domains ==
  -- example.com --
  -- example.org --
  -- domain-that-does-not-exist-xyz.test --
     (probe.sh reported a failure for domain-that-does-not-exist-xyz.test; continuing with the rest)
== Bringing back results with rsync ==
== Summary ==
-- Failures recorded --
      1 FAILED: no record resolves for 'domain-that-does-not-exist-xyz.test'
-- HTTP codes seen --
      2 http_code=200
Full results in: ./reports/20260721-221045
$ echo $?
0

The script exits with 0 even though a domain failed, because the report itself got generated correctly — it is the content of the summary, not remote-report.sh's exit code, that tells you something is down. That distinction between "the diagnosis failed" and "the diagnosis found a problem" is intentional and worth being able to explain.

README.md and how to defend it live

The README.md is the first thing someone who did not write the toolkit reads — including you yourself, six months later. At minimum, it needs the exit-code table (the same one you already used above, now in one single place), the requirements, and usage examples:

# ops-toolkit

Remote server diagnostics: local environment, a domain's network layers, and
orchestration via SSH with an aggregated report.

## Requirements

- bash 4 or newer
- `dig` (package `bind-utils` / `dnsutils`), `curl`, `ssh`, `rsync`
- `ss` (package `iproute2`) — only needed on the remote server, not on your laptop

## Environment variables

| Variable          | Used by         | Description                                |
|-------------------|-----------------|---------------------------------------------|
| `TOOLKIT_SSH_KEY`  | check-env.sh    | Path to the SSH private key to audit        |

## Exit codes

| Code   | Constant       | When it appears                                                | Script(s)                                |
|--------|----------------|------------------------------------------------------------------|--------------------------------------------|
| 0      | EX_OK          | Everything is fine                                                | all three                                   |
| 64     | EX_USAGE       | Invalid or missing argument                                       | all three                                   |
| 69     | EX_UNAVAILABLE | A command is missing / DNS does not resolve / check-env failed remotely | check-env.sh, probe.sh, remote-report.sh |
| 70     | EX_SOFTWARE    | The HTTP request failed (timeout, connection refused)             | probe.sh                                    |
| 77     | EX_NOPERM      | Unsafe permissions on the private key                             | check-env.sh                                |
| 78     | EX_CONFIG      | Missing env var / SSH alias / domains file                        | check-env.sh, remote-report.sh              |

## Usage

    ./check-env.sh --help
    ./probe.sh example.com --scheme https
    ./remote-report.sh webserver domains.txt

## How to demo it live

1. `./check-env.sh --help` and `./probe.sh --help` — shows the toolkit documents itself.
2. `./probe.sh example.com --dry-run` — shows the plan without touching the network.
3. Trigger a real failure (temporarily rename `dig`, or use a domain that does not exist) and
   show `echo $?` — the code matches the table above.
4. `./remote-report.sh <alias> domains.txt` against a list with one domain down on purpose —
   the final summary should show it without the whole script blowing up.
5. Open one of the three scripts and point to a specific decision: why that exit code and
   not another, what the `trap` cleans up, what would break if you removed `set -o pipefail`.

Defending this project live is not reading the code out loud — it is being able to answer three questions without looking at the screen: "why this exit code and not another?" (because you decided it and documented it, not because some law imposes it — more on this in common mistakes), "what happens if I run this without set -euo pipefail?" (a cd that fails silently, an empty variable treated as if it had a value, a failure in the middle of a pipe that nobody notices), and "what does the trap clean up?" (in remote-report.sh, the temporary folder on the server, never your local report). If you can answer those three without opening the file, you have already defended the project.

Common mistakes

Believing exit codes are a standard the system enforces. You defined 69 as "a dependency is missing" and 78 as "configuration is missing" — but that is a convention of yours, documented in your own README.md, not a law other programs respect. curl, for instance, has its own completely different table: code 6 means "could not resolve host" and 28 means "operation timed out" — nothing to do with the sysexits numbering this toolkit uses. How to spot it: if you chain your script's output with an external tool's and assume the same number means the same thing in both, you are going to misdiagnose the failure. How to fix it: never reuse an external program's exit code as if it were your own — check it in that program's own documentation (man curl has the full list) and decide yourself which of your own codes matches that case.

Trusting the first line of dig +short when the domain has a CNAME chain. A domain can resolve through several aliases before reaching an IP (www.example.com → cdn.example.net → 93.184.216.34), and dig +short prints the whole chain, one line per hop. Taking the first line with head -n1 gets you the first alias, not the final IP — and if something later in the script expects an IP address to, say, filter with ss, it is going to fail confusingly. How to spot it: run dig +short <domain> by hand and count how many lines come out; if there is more than one, the first one is almost never the IP. How to fix it: probe.sh uses tail -n1, which assumes the chain ends in an A record (the normal case) — for the general, robust case, filter explicitly with an IPv4 regular expression instead of trusting the line's position.

Writing remote-report.sh's loop without the safety || echo. Under set -euo pipefail, the first domain in the list that fails aborts the entire script right there — the domains after it in domains.txt never get tested, and you do not even find out they existed, because the final summary never gets generated either. It is the exact counterpart to the lesson-7 case where -e does not save you: here -e works exactly as designed (it aborts on a failure), but that behavior is the opposite of what this script needs for a batch of domains. How to spot it: run the toolkit against a list with one domain down in the middle and check whether the domains after it show up in the summary. How to fix it: any command inside a loop whose individual failure should not stop the rest needs its own explicit || — the "abort on the first error" policy is global unless you, command by command, decide otherwise.

Exercises

Exercise 1. check-env.sh does not check that ~/.ssh/config exists before remote-report.sh tries to read it later on. Add a check at the end of check-env.sh: if ~/.ssh/config does not exist or is not readable, exit with EX_CONFIG (78) and a clear message.

echo "== ~/.ssh/config =="
if [[ -r "${HOME}/.ssh/config" ]]; then
  echo "  ok: ${HOME}/.ssh/config exists and is readable"
else
  fail "${EX_CONFIG}" "${HOME}/.ssh/config does not exist or cannot be read (needed for remote-report.sh)"
fi

Exercise 2. Common mistake #2 points out that tail -n1 on dig +short is an assumption, not a guarantee. Rewrite that line in probe.sh so it specifically extracts the last IPv4 address from the output, no matter how many CNAME lines come before it.

IP="$(dig +short "${DOMAIN}" | grep -E '^[0-9]{1,3}(\.[0-9]{1,3}){3}$' | tail -n1 || true)"

Exercise 3. Add a third counter to remote-report.sh's summary: how many of the domains in domains.txt ended up with no FAILED line at all in their .err (total success) out of the total number of domains processed.

TOTAL=0
OK_COUNT=0
for f in "${LOCAL_REPORTS_DIR}"/*.err; do
  [[ -e "${f}" ]] || continue
  [[ "$(basename "${f}")" == "check-env.err" ]] && continue
  TOTAL=$((TOTAL + 1))
  [[ -s "${f}" ]] || OK_COUNT=$((OK_COUNT + 1))
done
echo "-- Domains with no failures: ${OK_COUNT}/${TOTAL} --"

Summary and next step

What you built in ops-toolkit is not an isolated exercise: it is the final form of everything you saw in this module — and, in a sense, in this entire guide. A script is no longer "a few lines to avoid typing the same thing twice," it is a piece with an explicit contract (documented arguments, --help, --dry-run, exit codes in a table) that someone else — or you, with less context, months later — can run without having to read the code first. That is the real boundary between a personal script and a tool.

You also touched lesson 7's limit: remote-report.sh already coordinates three scripts, two protocols (SSH and rsync over SSH), and its own report format. If tomorrow you also needed retries with backoff, parallel execution across several servers, or a JSON output format to feed a dashboard, that is exactly the point where it is worth rewriting it in Python — not because bash is "bad," but because you already used everything bash does well and what comes next is data structure, not more text.

Before considering this guide closed, you should be able to: write a script with its own argument parsing, a documented exit code per failure type, trap for cleanup, and a real (not cosmetic) --dry-run; diagnose a network problem explaining which layer you start with and why you rule out the others; and connect to a remote server to run something there and bring back the result, without needing to copy and paste commands by hand. If you are missing any of the three, go back to this module's lesson that covers it — you have the index above.

Resources

• curl(1) — write-out variables — the -w, --write-out section documents every variable like http_code, time_total, and time_connect that probe.sh uses.
• sysexits(3), OpenBSD manual — the origin of the exit-code convention (64, 69, 70, 78...) this toolkit adapts.
• ss(8), Linux manual — full syntax for ss filters, including the state-and-port variant used in probe.sh.
• ssh_config(5), OpenBSD manual — full reference for Host and its patterns, the basis for the validation remote-report.sh does.
• Google Shell Style Guide — a real organization's conventions for when to use shell, when not to, and how to structure scripts other people are going to maintain.