Module 3: Pipes, Redirection, and Composition

1. Introduction: commands that connect to each other

Next →

Description

Up to now you learned commands. Each one solves a specific question: ls tells you what is here, grep tells you where that word is, find tells you where that file lives. They are complete tools on their own, and they already got you out of real trouble in the previous two modules.

This module teaches you something different, and it is the exact point where the terminal stops being a list of loose commands and starts behaving like a language: you are going to connect one command's output to the next one's input, save results to files with surgical precision, and chain commands based on whether the previous one succeeded or failed instead of just assuming it did. By the end of the module you will be able to explain what standard input is and what any process's two outputs are — and why an error still shows up on screen even when you saved the output to a file — build a multi-command pipeline that answers a concrete question about a file with tens of thousands of lines, and decide with &&, ||, and ; which command runs after another.

The reason this matters in real work is simple: nobody who administers servers, reviews production logs, or automates a data flow does that work command by command, looking at the screen between each one and deciding by hand what to type next. They do it in one line that says exactly what to ask and where to get the answer from. That line is what this module teaches you to write.

Connection to the module: this lesson is the map: what you are going to be able to do, why this module changes how you work more than the previous two combined, and a warning about the order we are going to build it in. The next one, The Unix philosophy: small tools that compose, explains the design idea that makes everything that follows possible: why the Unix ecosystem bet on many small programs instead of one giant one, and what you gain — and what you lose — with that bet.

From loose commands to a language that composes

Think about how you learned a language. First, loose words: water, house, eat. With pure vocabulary you can already point at things and name them. But there is a huge distance between naming objects and saying "I want to eat something before getting home because there's no water there" — that sentence is not new vocabulary, it is grammar: the way of connecting words you already knew to express an idea none of them says on its own.

Modules 1 and 2 gave you vocabulary. ls, cd, find, grep, cat, wc: each one names something or does one specific thing. And they are useful exactly until the question in your head needs more than one of them to be answered. "Which IP requested something from my server the most times this week?" does not get answered by grep alone, or by sort alone — which you do not know yet: it gets answered by a sentence built from several commands, each one handing its result to the next.

That is this module's shift. You are not going to learn a more powerful command. You are going to learn that a command can receive something, transform it, and hand it to another one, and that chain can be as long as your question needs. The precise way of saying "every process has one input and produces two distinct kinds of output" is exactly what we are going to carefully define in lesson 3, and it is, without exaggeration, the concept you are going to use the most for the rest of your career working from a terminal.

Worked example: the question that gets answered in the time it takes to read it

Let us go to a concrete case, one you are going to run into again — at a bigger scale and with more questions — in the project that closes this module. Generate a web server log file yourself, access.log, with every line recording a request: which IP made it, to which path, and with what result. This loop is not new module content — it is the same for and the same case that exist in any shell; we only use it to manufacture realistic practice data, with a few IPs much more frequent than the others, the way it would happen with regular visitors or bots:

mkdir -p ~/pipes-preview && cd ~/pipes-preview
for i in $(seq 1 50000); do
  case $((i % 40)) in
    0|1|2|3|4|5|6) ip="203.0.113.45" ;;
    7|8|9|10|11) ip="203.0.113.12" ;;
    12|13|14|15) ip="198.51.100.7" ;;
    16|17) ip="198.51.100.23" ;;
    18) ip="203.0.113.201" ;;
    *) ip="10.0.$((i % 200)).$(((i / 200) % 200))" ;;
  esac
  echo "$ip - - [21/Jul/2026:12:00:00 +0000] \"GET /index.html HTTP/1.1\" 200 512"
done > access.log

Now, something you already know how to do since module 2 — measuring size before opening anything:

wc -l access.log

What to expect:

   50000 access.log

Fifty thousand lines. Open it with cat and your terminal is unusable for a good while. Open it in a spreadsheet and you first wait for it to load, then split the IP column with "Text to Columns," build a pivot table on that column, sort it largest to smallest, and only then look at the first rows. That back-and-forth of menus — load, split, build the table, sort, look — easily takes fifteen or twenty minutes for someone doing it for the first time, for a question that fits in a single sentence: which are the five IPs that requested something the most?

Here is that same question written for the terminal:

cut -d ' ' -f1 access.log | sort | uniq -c | sort -rn | head -5

What to expect:

8750 203.0.113.45
6250 203.0.113.12
5000 198.51.100.7
2500 198.51.100.23
1250 203.0.113.201

One line, and the answer shows up in the time it takes the terminal to process fifty thousand lines: a fraction of a second, on any modern machine. You do not need to understand that line yet. Look at it this way: it is five words you half-recognize, connected by a symbol you have not seen yet (|). cut, the first sort, uniq -c, and the second sort are new pieces — I introduce them to you in lesson 6 — but head you already know from module 2, and the idea that one tool's output becomes the next one's input is exactly what you are going to understand by heart two lessons from now. Come back to this line once you finish the module: you will be able to read it straight through and write a similar one for a question of your own.

Why the module starts backward from what you would expect

The logical thing, after seeing that example, would be to want to learn what that vertical bar is right now. Let us resist that temptation for one more lesson, and there is a concrete reason behind the order.

If you memorize that | "connects commands" and that > "saves to a file" without understanding what actually gets connected or saved, you memorized two spells. They work as long as you copy the exact example you studied, and they break the moment the situation changes a little — for example, when you save a failing command's output and the error still shows up on your screen, and you have no idea why, since you supposedly put > on it. That behavior is not a quirk or a bug: it is the direct consequence of a fact we have not explained yet, that a process does not have a single output, it has two, and they travel down different paths.

That is why lesson 3, The three streams: stdin, stdout, and stderr, does not teach you any new symbol. It gives you the model with which lessons 4 and 5's symbols stop being arbitrary. Once you understand that every process is born with one input and two separate outputs, > stops being "the save command" and becomes "point the results output somewhere else," and from there you can reason through syntax you have never seen instead of looking it up online every time you need it.

The module's map

1. Introduction: commands that connect to each other         ← you are here
2. The Unix philosophy: small tools that compose
3. The three streams: stdin, stdout, and stderr
4. Redirecting output to files: >, >>, 2>, and /dev/null
5. Pipes: connecting one command's output to another's input
6. The text toolkit: sort, uniq, cut, tr, and sed
7. Exit codes and chaining: $?, &&, ||, and ;
8. Project: a log analysis pipeline

Notice the shape: two lessons of theory (2 and 3) before touching a single symbol, four lessons of mechanics (4 through 7), and a project (8) that pulls it all together against a real log — the same kind of file you opened in the example above, but with more questions and no one handing you the answer on a plate.

Common mistakes

Confusing "knowing many commands" with "knowing how to solve problems in the terminal." It is easy to arrive at this module thinking what is missing is memorizing command number twenty-one, more advanced than the previous ones. That is not it: what is missing is the grammar that connects the vocabulary you already have. You spot it when, faced with a question like the one in the example above, your first instinct is to open the file in an editor and eyeball it instead of thinking of a chain of commands. You fix it with deliberate composition practice — exactly what this module trains — rather than by piling up more loose commands.

Wanting to skip lesson 3's theory and go straight to | and >. You are going to feel the temptation, especially after the example above. The problem is that, without the model of two separate outputs, you are going to memorize the syntax instead of understanding it, and that kind of memory fades in a couple of weeks — you notice it because you have to look up online something you already used last month. You fix it by respecting the module's order this one time; afterward you are not going to need the explicit order anymore, because the mental model will already be in place.

Building a long pipeline all at once, without testing each link. Once you finally know how to write something like the example above, the temptation is going to be to write all five commands in one go and run everything together. If something goes wrong — empty output, a result that makes no sense — you are not going to know which of the five links failed. You spot it because the error or the silence does not tell you which step it happened at. You fix it by building one command at a time, checking the result at each step, and adding the next one only once the previous one does what you expected — it is the full method from lesson 5, but it is worth starting the habit from now.

Exercises

Exercise 1 — Spreadsheet or terminal?

For each scenario, decide whether you would solve the question with a spreadsheet, with the terminal, or combining the two, and justify it with what you saw in this lesson: speed, the file's scale, and what you need the result for afterward.

  1. A 12-row file with the quarter's sales, and you need to build a bar chart for a meeting in five minutes.
  2. An 80,000-line server log, and during a production incident you need to know right now which are the five most frequent error codes.
  3. A 40,000-line transaction file you need to filter down to just one client's, count how many there were per month, and hand the result to a manager expecting a tidy table with a chart.
See solution
  1. Spreadsheet. Twelve rows is a scale with no real friction to opening the file, and you need a chart — something a spreadsheet does much better than the terminal. None of this module's advantages, speed over huge files and step composition, apply here.
  2. Terminal. Eighty thousand lines and an urgent question during an incident is exactly the lesson's example case: opening that in a spreadsheet means waiting for it to load and building a pivot table while the problem keeps happening. A command line gives you the count in seconds.
  3. Both, combined. Filtering and counting across 40,000 lines is terminal work — fast and precise — but "tidy table with a chart for a manager" is exactly what a spreadsheet does well and the terminal does not. The real pattern, and the one you are going to use at work, is processing in the terminal and presenting in the spreadsheet. It is not a competition between the two tools.

Why it works: the criterion is not "the terminal always wins." It is evaluating the file's scale, how urgent the answer is, and what needs to happen with the result afterward — the same three factors that separated the lesson's example from a case where the spreadsheet is still the right tool.

Exercise 2 — Read before running

You have not formally learned cut, sort, or uniq yet — they arrive in lesson 6 — but you already know something module 1 taught you: read a manual before using a command. Without running anything else, run these three commands and, based on what you see on each page, write in one sentence what you think each tool does:

man cut
man sort
man uniq
See solution

There is no single correct wording, but the meaning should come close to this:

  • cut trims a piece out of every line of a file — for example, a column, when the values are separated by a character like a space or a comma.
  • sort sorts a file's or an input's lines, alphabetically by default.
  • uniq removes consecutive repeated lines, and can also count how many times each one repeated.

Why it works: each manual's short description — the NAME section, right below the title — sums up in one line what the command does, without you needing to understand its flags yet. It is the same skill from module 1's lesson 5, reading before running, now applied to commands you are going to formally learn shortly. If you connect these three sentences to this lesson's example line (cut ... | sort | uniq -c | sort -rn | head -5), you can already guess what each link does, even though you cannot write one yourself yet.

Exercise 3 — Apply the composition lens

You have to answer this question about your own machine: "how many .log files were modified in the last 7 days inside /var/log?" You already know find from module 2, which can filter by extension and by modification date. With what you saw today: does find alone give you the final answer, or are you going to need to combine it with something else? Justify it.

See solution

find can filter and list the files meeting that condition, but what it hands you is a list of paths, one per line, not a number. To get to "how many," you need to combine it with something that counts lines. You already know that piece: wc -l, from module 2. The full question, then, does not get answered by a single command: it gets answered by a composition of two — exactly the kind of reasoning this module formalizes, even though you have not seen the exact syntax for connecting them, the pipe, yet.

Why it works: recognizing that a question has two parts — "finding" and "counting" — and that each part already has a known command assigned to it is this module's central mental habit. The syntax for joining them is mechanical and you learn it in lesson 5; the judgment to know you need to join two commands is what you are training right now.

Summary and next step

Modules 1 and 2 gave you commands that solve specific questions. This module gives you the grammar to connect them: one's output becomes the next one's input, you can decide with precision what to save and what to discard, and you can chain commands based on whether the previous one succeeded or failed. You saw that this shift is not cosmetic — a question that takes five steps and half an hour in a spreadsheet is, in the terminal, a line you read straight through in a couple of seconds — and why the module flips the usual order: theory, the three streams, before symbols, so that what you learn actually sticks.

Before moving on you should be able to:

  • Explain in your own words why "knowing many commands" is not the same as "knowing how to connect them" to solve a question none of them answers alone.
  • Name, without looking at the map, the idea lesson 3 is going to precisely define: one input and two distinct outputs for every process.
  • Read this lesson's example line (cut ... | sort | uniq -c | sort -rn | head -5) and say out loud, even approximately, what each link does.
  • Say why this module teaches theory before syntax, the reverse of what modules 1 and 2 did.

What comes next is not syntax yet. It is the design idea that explains why small commands exist instead of one giant one that does everything, and why that decision, made in the 1970s, is still the reason you can build the line above with pieces nobody designed with each other in mind.

Resources

← Previous8. Project: audit a project you do not knowNext →2. The Unix philosophy: small tools that compose