8. Project: a log analysis pipeline

Description

In this capsule you are going to build, from scratch, an analysis pipeline capable of reading a server log file with tens of thousands of lines and answering five real traffic questions — which are the most active IPs, how the status codes break down, which is the peak hour, which routes are failing the most, and what percentage of requests end in an error — each in a single command line. You are going to save that report with tee, separate real errors with 2>, and chain each step with && so the report never gets generated from data you could not read.

This is, almost word for word, what an on-call engineer does when an alert goes off at three in the morning: there is no time to write a script, there is not always access to install a new tool, and the monitoring system is sometimes down right when it is needed most. The only thing that is always there is the terminal and the raw log file. Knowing how to turn "what is going on?" into a pipeline of commands in under a minute is a skill that stands out in a technical interview and stands out even more during a real incident.

Connection to the module: this capsule does not teach any new command — it pulls together everything you saw in the previous seven. cut, sort, and uniq (capsule 6) do the heavy lifting; >, 2>, and /dev/null (capsule 4) separate what you want to keep from what you want to discard; | (capsule 5) connects each piece to the next; and &&, $?, and grouping with parentheses (capsule 7) decide whether the report finishes building or stops in time. If any of those pieces feels shaky, this is the capsule where it shows.

From question to command: thinking in pipelines

Imagine an assembly line: one station welds, the next paints, the next packs. Every station does exactly one thing and the piece moves along a conveyor belt. If the welding station detects a defective piece, it makes no sense for it to keep going to paint and packing — the belt stops right there, before wasting paint and boxes on something that was already wrong. A log analysis pipeline is exactly that assembly line, except the stations are called cut, sort, uniq, and grep, the conveyor belt is |, and what decides whether the line keeps going or stops is the exit code you already know from the previous capsule.

For the stations to work, you first need to understand the shape of the raw material: each line of the log. A typical web server writes its lines in a format called Common Log Format, with fields separated by spaces in a fixed order. This real line from our practice file:

8.84.176.107 - - [21/Jul/2026:22:20:01 +0000] "DELETE /index.html HTTP/1.1" 500 2608

splits by spaces into ten fields:

Notice fields 4 and 5: the full date has a space inside it (before the time zone), so cut — which only knows how to cut by the literal delimiter you give it, not by "where a logical piece of data starts and ends" — splits it into two separate fields. You are going to use exactly this quirk to extract the hour in question 3.

Worked example

The log file

You need a file with tens of thousands of lines for the questions to make sense — with five lines any answer is trivial. If your bootcamp or team already gave you a real access.log, use it. If not, this command manufactures one with 50,000 realistic-looking lines (IPs that repeat with different frequencies, as would happen with real visitors and bots, and one route — /api/orders — with more 500 errors than the others, on purpose, so question 4 has a clear answer).

You do not need to understand every detail of this command — it uses awk, a tool that is not part of this module. Copy it as-is, just to manufacture test data:

awk 'BEGIN {
  srand(42)
  split("GET GET GET GET POST PUT DELETE", methods, " ")
  split("/index.html /login /api/users /api/orders /images/logo.png /cart /checkout /api/products /favicon.ico /about", paths, " ")
  split("200 200 200 200 200 301 302 404 404 500", statuses, " ")
  n_ips = 300
  for (i = 1; i <= n_ips; i++)
    ip_pool[i] = int(rand()*223)+1 "." int(rand()*255) "." int(rand()*255) "." int(rand()*255)
  for (i = 1; i <= 50000; i++) {
    idx = (rand() < 0.15) ? int(rand()*5)+1 : int(rand()*n_ips)+1
    ip = ip_pool[idx]
    hour = sprintf("%02d", int(rand()*24)); min = sprintf("%02d", int(rand()*60)); sec = sprintf("%02d", int(rand()*60))
    method = methods[int(rand()*7)+1]; path = paths[int(rand()*10)+1]; status = statuses[int(rand()*10)+1]
    if (path == "/api/orders" && rand() < 0.35) status = "500"
    bytes = int(rand()*5000)+200
    printf "%s - - [21/Jul/2026:%s:%s:%s +0000] \"%s %s HTTP/1.1\" %s %s\n", ip, hour, min, sec, method, path, status, bytes
  }
}' > access.log

wc -l access.log

What to expect:

   50000 access.log

The exact numbers you will see in the rest of this capsule (which IP ends up first, how many 500s there are) are the ones this generator produced on this run. If you run the same command on your machine, you will almost certainly get different numbers — every awk implementation (macOS's is not the same as Ubuntu's) builds its own sequence of "random" numbers, even with the same seed. That is normal and not an error: what matters is that you reproduce each pipeline's shape, not the exact digit.

Question 1: the 10 IPs with the most requests

cut -d' ' -f1 access.log | sort | uniq -c | sort -rn | head -n 10

• cut -d' ' -f1 keeps only field 1 of every line (the IP).
• sort groups identical IPs next to each other — without this, the next step is useless (you will see this in Common Mistakes).
• uniq -c collapses consecutive repeated lines and prefixes how many times each one showed up.
• sort -rn reorders those counts largest to smallest (-n numeric, -r reversed).
• head -n 10 keeps the first 10 lines.

What to expect:

1702 68.6.93.195
1688 8.84.176.107
1675 46.63.162.220
1664 19.140.143.189
1607 71.34.27.193
 169 24.179.72.128
 167 77.182.209.36
 166 67.75.77.224
 166 3.167.184.216
 166 14.9.22.104

Notice the jump between the fifth line (1607) and the sixth (169): those are the five "bots" the generator concentrated on purpose. On a real log, a jump like that is almost always a signal — a scraper, a brute-force attack, or a misconfigured client retrying nonstop.

Question 2: requests by status code

With the log's anatomy already mapped, field 9 is the status:

cut -d' ' -f9 access.log | sort | uniq -c | sort -rn

What to expect:

23885 200
9803 404
6671 500
4865 302
4776 301

Same pipeline as question 1 — only the field number changes. That is this module's underlying idea: you do not memorize a new command for every question, you recombine the same four.

Question 3: the busiest hour

This is where field 4's quirk from above comes in. cut -d' ' -f4 leaves you with [21/Jul/2026:HH:MM:SS (with no time zone) — still not the hour alone. A second cut, this time with : as the delimiter, isolates it:

cut -d' ' -f4 access.log | cut -d: -f2 | sort | uniq -c | sort -rn | head -n 1

Break it down: [21/Jul/2026:22:20:01 split by : gives fields [21/Jul/2026, 22, 20, 01 — field 2 is the hour.

What to expect:

2135 07

Hour 07 had 2,135 requests — just barely above the rest (with traffic spread almost randomly, the "winner" wins by a little). On a real production log, with human usage patterns, this difference is usually much more pronounced.

Question 4: the routes with the most 500 errors

This question needs two filters: keeping only the lines whose status is 500, and from those, extracting the route (field 7, but inside the quotes of field 6-8). You use grep for the first part — searching for the exact pattern that separates the status from the rest of the line — and two cuts with different delimiters for the second:

grep 'HTTP/1.1" 500 ' access.log | cut -d'"' -f2 | cut -d' ' -f2 | sort | uniq -c | sort -rn | head -n 10

• grep 'HTTP/1.1" 500 ' keeps only the lines where the status is 500 (the trailing space prevents "5000" or "50000" from being a false positive).
• cut -d'"' -f2 cuts by quotes: from "DELETE /index.html HTTP/1.1" it keeps the second chunk, DELETE /index.html HTTP/1.1.
• cut -d' ' -f2 cuts that chunk by spaces and keeps field 2: the route.

What to expect:

2045 /api/orders
 540 /api/products
 537 /index.html
 526 /images/logo.png
 521 /checkout
 516 /about
 515 /favicon.ico
 505 /api/users
 487 /login
 479 /cart

/api/orders has almost four times more 500 errors than the next route. In a real incident, this line is exactly the piece of data that tells you where to look first: not "the site is slow," but "the orders endpoint is returning server errors."

Question 5: percentage of failed requests

For the percentage you need two counts (total and failed) and a division with decimals — something the terminal does not do with integers. bc is a calculator that reads an expression from its standard input and writes the result to its standard output: the same input/output pattern from this whole module, applied to arithmetic instead of text.

TOTAL=$(wc -l < access.log)
FAILED=$(grep -cE 'HTTP/1.1" [45][0-9][0-9] ' access.log)
echo "scale=2; $FAILED * 100 / $TOTAL" | bc

• wc -l < access.log counts the total lines (the < redirection instead of passing the file's name keeps wc from printing the name next to the number).
• grep -cE 'HTTP/1.1" [45][0-9][0-9] ' counts (-c) how many lines have a status starting with 4 or 5 (-E enables the [45] class as an extended regular expression) — that is, client or server errors. The 3xx (redirects) do not count as failed.
• scale=2 tells bc to show two decimal places; without it, bc works with integers by default and the result would get truncated to 0.

What to expect:

32.94

Almost a third of this log's requests ended in an error — a number that on a real server would trigger an alert immediately.

The deliverable: a chained script

Five standalone pipelines are fine for exploring, but this project's deliverable is a single file that runs them all, in order, stopping the moment something goes wrong — and that separates real errors from results with 2>. Save this as analyze-log.sh:

#!/usr/bin/env bash
# analyze-log.sh
# Traffic report built from access.log.
# Run with: bash analyze-log.sh

LOG="access.log"

# Guard: if the file does not exist, fail right here -- with the real
# error from "ls" visible -- before spending a single analysis command.
# Expected exit code: 0 if the file exists, 1 if not.
ls "$LOG" > /dev/null 2> errors.log &&

# Question 1: the 10 IPs with the most requests.
# Expected exit code: 0
cut -d' ' -f1 "$LOG" 2>> errors.log | sort | uniq -c | sort -rn | head -n 10 > top-ips.txt &&

# Question 2: requests by status code (field 9).
# Expected exit code: 0
cut -d' ' -f9 "$LOG" 2>> errors.log | sort | uniq -c | sort -rn > status-codes.txt &&

# Question 3: the busiest hour.
# Expected exit code: 0
cut -d' ' -f4 "$LOG" 2>> errors.log | cut -d: -f2 | sort | uniq -c | sort -rn | head -n 1 > busiest-hour.txt &&

# Question 4: routes with the most 500 errors.
# grep returns 1 if it does NOT find any 500 -- that is not a real
# failure, so "|| true" keeps it from cutting the chain in that case.
(grep 'HTTP/1.1" 500 ' "$LOG" 2>> errors.log | cut -d'"' -f2 | cut -d' ' -f2 | sort | uniq -c | sort -rn | head -n 10 > top-500-routes.txt || true) &&

# Question 5: percentage of failed requests (4xx and 5xx).
TOTAL=$(wc -l < "$LOG") &&
FAILED=$(grep -cE 'HTTP/1.1" [45][0-9][0-9] ' "$LOG" 2>> errors.log) &&
echo "scale=2; $FAILED * 100 / $TOTAL" | bc > failure-rate.txt &&

# Final link: builds the complete report. "tee" saves it to report.txt
# AND ALSO lets it pass through to the screen -- that is why "tee" and not ">".
cat top-ips.txt status-codes.txt busiest-hour.txt top-500-routes.txt failure-rate.txt | tee report.txt

STATUS=$?
echo "Exit code of the last link: $STATUS"
exit "$STATUS"

Run it with bash analyze-log.sh.

What to expect (with access.log present):

1702 68.6.93.195
1688 8.84.176.107
...
32.94
Exit code of the last link: 0

And the directory ends up with top-ips.txt, status-codes.txt, busiest-hour.txt, top-500-routes.txt, failure-rate.txt, report.txt (the same content you saw on screen) and an empty errors.log.

What to expect if access.log does not exist (for example, if you misspelled the name in the LOG variable):

Exit code of the last link: 1

errors.log contains a single line (ls: access.log: No such file or directory), and no other file got created — no half-built top-ips.txt, no incomplete report.txt. The chain stopped at the first link, exactly as the project's goal asked for.

When even ls is not enough: PIPESTATUS

The ls trick works because you can check the problem before starting the analysis. But sometimes the command that can fail is already part of the pipeline you need to run — there is no way to "check it beforehand." For those cases, bash stores every link's exit code (not just the last one) in an array called PIPESTATUS:

cut -d' ' -f1 does-not-exist.log | sort | uniq -c > /dev/null
echo "${PIPESTATUS[@]}"

What to expect:

cut: does-not-exist.log: No such file or directory
1 0 0

The first number is cut's exit code (1, it failed), the second is sort's (0), the third is uniq -c's (0). With this you know exactly which of the three links failed, not just whether the last one succeeded. If you use zsh instead of bash, the same array exists under the name pipestatus (lowercase).

An alternative, if all you care about is knowing "did anything in this pipeline fail?" regardless of which one, is the set -o pipefail option: from then on, any pipeline's $? reflects the code of the last command that failed (not necessarily the last one on the line), instead of always reflecting the last link no matter what happened before it.

Common mistakes

1. Using uniq -c with no sort before it (conceptual). uniq does not remove global duplicates: it only collapses consecutive identical lines. If the repeated IPs are scattered through the file (normal in a real log, where the same client comes back minutes later), uniq -c with no sort before it counts them as separate one-line groups. Compare:

cut -d' ' -f1 access.log | uniq -c | sort -rn | head -n 3

   3 8.84.176.107
   3 71.34.27.193
   3 68.6.93.195

against the correct version:

cut -d' ' -f1 access.log | sort | uniq -c | sort -rn | head -n 3

1702 68.6.93.195
1688 8.84.176.107
1675 46.63.162.220

Without sort, the "most repeated" IP looks like it has 3 requests instead of 1,702, and the whole file appears to have 49,618 distinct IPs instead of 300. You spot it because the numbers do not add up against the file's total line count; you fix it by always putting sort immediately before uniq -c.

2. Confusing field 4 with the full date. cut -d' ' -f4 access.log does not give you [21/Jul/2026:22:20:01 +0000] — it gives you only [21/Jul/2026:22:20:01, because there is a space before the time zone and cut blindly cuts on that space, with no idea both chunks are "logically" a single date. You spot it when a cut's result looks truncated in the middle of something you expected whole; you fix it by checking how many space-separated fields the data actually spans (here it is two: field 4 and field 5).

3. Trusting a pipeline's $? to know whether a command in the middle failed. By default, a pipeline's exit code is the last command's, no matter what happened before:

cut -d' ' -f1 does-not-exist.log | sort | uniq -c | sort -rn | head -n 3
echo "Exit code: $?"

cut: does-not-exist.log: No such file or directory
Exit code: 0

cut failed and said so on stderr, but sort, uniq -c, sort -rn, and head received an empty input and "succeeded" processing it — so the full pipeline reports 0. An && depending on that 0 would move right along as if everything were fine. You spot it when a report comes out "empty but with no visible error"; you fix it by checking errors.log (or $PIPESTATUS, seen above) instead of trusting only the pipeline's final exit code.

Exercises

1. Using access.log, write a single command line that shows the 5 most-requested routes overall (regardless of status code), sorted largest to smallest.

cut -d'"' -f2 access.log | cut -d' ' -f2 | sort | uniq -c | sort -rn | head -n 5

2. Write a command line that counts how many requests each HTTP method made (GET, POST, PUT, DELETE).

cut -d'"' -f2 access.log | cut -d' ' -f1 | sort | uniq -c | sort -rn

3. Calculate what percentage of requests had a success code (2xx), reusing question 5's pattern.

TOTAL=$(wc -l < access.log)
OK=$(grep -cE 'HTTP/1.1" 2[0-9][0-9] ' access.log)
echo "scale=2; $OK * 100 / $TOTAL" | bc

4. You want a command line that saves the top 10 IPs into top-ips.txt and also shows it on screen, but only if access.log exists. If it does not exist, you do not want an empty top-ips.txt or a confusing message — you want the only visible message to be the real error that the file is missing. Write it and test it pointing at a file that does not exist.

ls access.log > /dev/null && cut -d' ' -f1 access.log | sort | uniq -c | sort -rn | head -n 10 | tee top-ips.txt

Summary and next step

What you just built is not a one-off trick: it is the same move — filter, sort, count, chain based on the result — that you are going to repeat in front of any large text file for the rest of your career, whether it is a server log, a CSV exported from a database, or another program's output. Before moving on you should be able to, without looking at this capsule: explain why sort always has to come before uniq -c; predict, before running a pipeline with &&, whether a failure halfway through is going to stop it or let it through; and build from memory a three- or four-command pipeline to answer a new question about a text file you have never seen.

Everything you did today you typed command by command, by hand, in the terminal. A shell script is exactly this — commands chained with &&, exit codes checked, errors separated from results — saved to a file, with variables and control structures so you do not repeat the same logic twice. That is what opens module 5.

Resources

• Log Files — Apache HTTP Server — official documentation for the Common Log Format used in this project.
• cut invocation — GNU Coreutils Manual — full reference for cut's options, including -d and -f.
• sort invocation — GNU Coreutils Manual — all sorting options, including -r and -n.
• uniq invocation — GNU Coreutils Manual — why uniq requires sorted input and what -c does.
• Pipelines — Bash Reference Manual — where a pipeline's exit code comes from and what pipefail changes.
• tee(1) — Linux manual page — full reference for the tee command.