Module 3: Pipes, Redirection, and Composition

3. The three streams: stdin, stdout, and stderr

Description

Imagine this scene: you leave a backup script running overnight and save its output to a log file to check in the morning. The next day you open that file and it looks perfect — not a single line of error. But the backup never happened. How can a log file be clean and the job have failed anyway? The answer is not in the script: it is in something every process has from the instant it starts, and that you have used without naming so far.

In this lesson you will be able to explain why a program has three separate communication pipes — one input and two outputs — identify which of those three any message you see in your terminal traveled through, and diagnose the case above: why an error still shows up on screen (or vanishes without a trace) even though you saved "the output" of the program. This is not a technical curiosity. It is why a data pipeline, an automated deployment, or a cron job can fail silently: if you do not know two separate outputs exist, you capture one and assume you captured both.

Connection to the module: in the previous lesson you saw that every Unix program produces plain text so another program can consume it — the universal interface that makes composition possible. What was missing to nail down is that this text does not come out of a single place: it comes out of two, and each one has a different job. The numbers you are going to meet here (0, 1, and 2) are not trivia: they are literally the syntax you are going to write in the next lesson (2>, 1>, &>). Without this mental model, that syntax gets memorized like a spell; with it, it gets deduced.

A process is born with one input and two outputs

Think of a process as if it were an assistant sitting at a desk with three baskets. One input basket, where instructions or data to work on arrive. And two output baskets — not one: two. In the first they put the finished result, what you asked for. In the second they put something different: a separate note when something did not go well, a file they could not find, a permission they were missing. They never mix the two output baskets together, even though both sit on the same desk and, if you do not separate them yourself, both end up in front of your eyes on the same screen.

That is exactly what a process does in the terminal. When any program starts, the operating system automatically opens three channels for it, with no need for the program to ask:

Name	Technical name	Number (descriptor)	What it is for
Standard input	`stdin`	`0`	Where the program reads from what it needs to process (normally, what you type on the keyboard)
Standard output	`stdout`	`1`	Where the program writes its normal result — what you asked for
Standard error	`stderr`	`2`	Where the program writes its complaints: errors, warnings, diagnostic messages

As a diagram, the shape of any process is always this:

   standard input                 ┌─────────────┐
   (stdin, descriptor 0) ───────► │             │
   normally: your keyboard        │   process   │
                                  │             │──────► standard output
                                  │             │        (stdout, descriptor 1)
                                  │             │        the result you asked for
                                  │             │
                                  │             │──────► standard error
                                  └─────────────┘        (stderr, descriptor 2)
                                                          complaints and diagnostics

Those numbers — 0, 1, and 2 — are not a decorative convention: they are the file descriptor the operating system assigns to each channel. A file descriptor is just a number the system uses to identify an open input or output channel; the first three any process receives, without exception, are always these same ones. Memorize the numbers now, even though you are not doing anything with them yet: in the next lesson you are going to literally write 2> to say "descriptor 2, redirect it over there," and 1> (almost always abbreviated >) to say the same about descriptor 1. Tomorrow's syntax is, letter for letter, this table.

One piece of context that helps this feel less arbitrary: stderr did not always exist. On the earliest Unix systems in the 1970s, errors got mixed in with normal output in a single stream, and on the machines of the time that literally wasted paper: typesetting machines printed error messages as if they were part of the result. Separating a third channel just for diagnostics was the fix, and it stuck forever as part of Unix's design.

Worked example

Let us see it, not just read about it. Run this command as-is — you do not need to create any file, /etc/hosts exists on macOS, Linux, and WSL — asking ls to list a real file alongside one that does not exist:

ls /etc/hosts /etc/hosts-fantasma

What to expect: on your screen you are going to see two lines mixed together, something like this:

ls: /etc/hosts-fantasma: No such file or directory
/etc/hosts

Notice a detail that is not a coincidence: in the command you typed the existing path first and the nonexistent one second, but on screen the error shows up first. That is direct evidence that this is not a single stream of text in order: they are two independent channels, each delivered to the screen at its own pace, and they only line up there because nobody told them otherwise yet.

Now the part that answers the introduction's question. Save only the normal result to a file, with > (the symbol you are going to study in depth next lesson; for now you just need to know it tells the shell "whatever comes out of descriptor 1, instead of showing it on screen, write it to this file"):

ls /etc/hosts /etc/hosts-fantasma > listing.txt

What to expect: on screen you see only this — the error, and nothing else:

ls: /etc/hosts-fantasma: No such file or directory

And if you open the file you just created:

cat listing.txt

/etc/hosts

There is the answer to the overnight-backup scenario. The file came out spotless — it contains exactly the clean result, not a single complaint — because > only grabs descriptor 1. The error never had anywhere to go but the screen, and if nobody was watching that screen at three in the morning (for example, because the script ran as a cron job with no one having a session open), that message got lost with no trace in any file. A clean log did not mean everything went well: it meant you only captured one of the two outputs.

Why this separation exists

The underlying reason is exactly what you just verified: separating the channels lets you decide, with precision, what to do with each one. If stdout and stderr were a single stream, there would be no way to save "the clean result" without any complaint sneaking in the middle — and there would also be no way to silence just the complaints without also losing the result you actually needed. A data analysis pipeline expecting a well-formed CSV file breaks the instant an error line sneaks in among the data; an automated process that needs to know whether something failed goes blind if that signal gets lost among the normal result.

This is exactly the problem having two separate outputs by design solves: you can capture the clean result in a file and at the same time let the complaints stay visible (or send them elsewhere, or discard them on purpose). It is not a coincidence or over-engineering: it is the only way for "save the result" and "tell me if something goes wrong" to be two independent decisions instead of one forced decision. The next lesson gives you exactly that control — > for descriptor 1, 2> for descriptor 2, and the ways to combine them — but the control only makes sense if you understood first, as you just did, that there are two distinct channels to control.

Common mistakes

"If I did not see red text, there was no error." That many terminals and tools (like Git or some linters) color stderr red is a visual courtesy from that particular tool, not a property of stderr. Descriptor 2 carries no color: it is a channel, not a format. There are systems and configurations where stdout and stderr look exactly the same — white text on black, no distinction — and the error goes unnoticed if you only look for its color. The correct way to verify is structural, not visual: like you did in the example, redirect one channel and observe what disappears from the screen and what stays.

A command that "freezes" and is actually waiting for your keyboard. Type cat with no arguments and press Enter. The terminal shows no new prompt, the cursor blinks, and it feels like something broke. Nothing broke: cat with no file to read falls back to its standard input by default, which normally is your keyboard, and sits there waiting for you to send it something. It is not an error, it is stdin's documented behavior. You spot it because there is no error message or returned prompt, just silence; you fix it by typing something and pressing Ctrl+D at the start of an empty line, which tells the program "end of input" (technically, it sends EOF — end of file) and makes it finish with whatever you gave it. Ctrl+C also gets it out of the block, but by aborting the command instead of letting it process what you typed.

"The log file came out clean, so the program finished fine." This is the mistake that opened this lesson, and it is worth calling out separately because it is the most expensive one in production: a clean log file only proves descriptor 1 reported no problems. It says nothing about descriptor 2, unless you also explicitly pointed it at that same file (something you are going to learn to do with 2>&1 in the next lesson). Before trusting "the log has no errors" as proof of success, first check where each of the two channels was pointing when the program ran.

Exercises

1. Identify the channel

Without running anything yet, look at this command and its on-screen output:

wc -l /etc/hosts /etc/passwd-fantasma

wc: /etc/passwd-fantasma: open: No such file or directory
       12 /etc/hosts
       12 total

Which line or lines traveled through stdout and which one traveled through stderr? Justify your answer with what each line represents, not just its position.

See solution

The line wc: /etc/passwd-fantasma: open: No such file or directory traveled through stderr (descriptor 2): it is a diagnostic message about something that went wrong (a file wc could not open), not a result. The lines 12 /etc/hosts and 12 total traveled through stdout (descriptor 1): they are the normal result you asked for — the line count of the file that does exist, and the total.

Why it works: the criterion for classifying a line is never its position on screen or its format — it is what it represents. An expected result (a count, a listing, a piece of data) is stdout; a message about a problem (a missing file, a denied permission) is stderr, no matter what order they show up mixed together in.

2. The command that does not respond

A coworker writes to you, worried: "I typed grep pending in the terminal, pressed Enter, and now it does nothing. No error shows up, the prompt that lets me type another command does not come back. Did my terminal freeze?" What is actually going on, and what do you tell them to do?

See solution

Nothing froze. grep pattern with no file name as a second argument has nowhere to read text from to search for the pattern, so it falls back to its standard input by default: the keyboard. It sat there waiting for someone to type lines of text right there, in the terminal, to search for pending in whatever comes in. It is exactly the same behavior you saw with cat with no arguments, just with grep instead of cat.

Tell them to type whatever they want (for example, a few lines with and without the word "pending") and press Ctrl+D at the start of an empty line to tell grep the input is done; at that point it is going to print the lines that did contain the pattern and hand back control of the terminal. If they just want to quit without using it, Ctrl+C aborts the command without processing anything.

Why it works: a program waiting silently, with no prompt or error message, is almost always reading from stdin and nobody gave it a file — it is not a sign of failure, it is the absence of the argument that would have prevented that wait.

3. Predict the syntax before seeing it

You already know descriptor 1 is stdout and descriptor 2 is stderr, and that > moves descriptor 1 to a file. Without looking anything up yet: what do you think the syntax 2> errors.txt does? Do not run it — write your hypothesis first, in your own words.

See solution

2> errors.txt should redirect descriptor 2 (stderr) to the errors.txt file, letting descriptor 1 (stdout) keep printing on screen as usual — exactly the mirror of what you did in the worked example with > (which is the short form of 1>).

Why it works: redirection syntax is not a set of symbols to memorize separately; it is a descriptor's number followed by an arrow to a destination. If you understood that 1 and 2 are two distinct channels, each one's syntax is predictable before reading the documentation — which is exactly what you are going to confirm in the next lesson.

4. The order that was not a coincidence

In the worked example, the command was ls /etc/hosts /etc/hosts-fantasma, with the real path written first. And yet, on screen the error message showed up before the real file's name. Why can this happen, if the error corresponds to the command's second argument?

See solution

Because stdout and stderr are two independent channels, each with its own delivery behavior to the screen (known as buffering): stderr is usually delivered right away, line by line, while stdout can build up for an instant before being shown. The order you typed the arguments in the command does not determine the order their effects reach your screen, precisely because they do not share a single output pipe.

Why it works: if stdout and stderr were the same stream, the order they appear on screen would always match the order the program generated them in. That it does not match is observable proof — not just theoretical — that they travel through separate channels.

Summary and next step

Before moving on you should be able to:

name any process's three streams (stdin, stdout, stderr) and their descriptors (0, 1, 2) without hesitating;
look at mixed output on screen and classify each line by what it represents, not by its position;
recognize a program waiting for data on stdin (silence, no prompt, no error) and know Ctrl+D ends the input while Ctrl+C aborts the command;
explain why a "clean" log file is not proof everything went well, if you only captured descriptor 1.

What you just installed is the full mental model that makes this module's next four lessons obvious. The next one takes exactly the numbers 0, 1, and 2 you just met and gives them their real syntax: > and 1> to redirect the result, 2> to redirect the error, 2>&1 and &> to join both on purpose, /dev/null to discard what you do not care about, < to feed input from a file instead of the keyboard, and tee to see something on screen and save it at the same time. You are not going to memorize loose symbols: you are going to recognize, in each one, the same descriptor you already know how to name.

Resources

stdin(3) — Linux manual page — the technical reference for stdin, stdout, and stderr, with explicit confirmation that their descriptors are 0, 1, and 2 when any program starts.
The Open Group Base Specifications — stdin — the POSIX standard defining the three predefined streams and the STDIN_FILENO, STDOUT_FILENO, and STDERR_FILENO constants.
Standard streams — Wikipedia — a general overview of the three streams, including the historical reason Unix separated stderr from stdout in the 1970s.
Making cron notify only on failure — James Cherti — a real case for why separating stdout from stderr matters in automation: cron emails you anything a job writes to either of the two channels.

← Previous2. The Unix philosophy: small tools that compose Next →4. Redirecting output to files: >, >>, 2>, and /dev/null