5. Pipes: connecting one command's output to another's input

Description

By the end of this lesson you will be able to build pipelines of several commands — two, three, four links — to answer questions about files you could never review by hand: how many error lines are in a log with tens of thousands of rows, how many files of a certain type exist across the whole system, which of a search's thousands of results are worth a closer look. You are going to build them with a concrete method — piece by piece, verifying every link — and you are going to know what to do when the next command in the chain refuses to read what you send it.

This is exactly what a developer or a systems administrator does when a server fails at three in the morning: nobody opens a two-million-line log file in a text editor to eyeball it. Three or four small commands get connected, each one doing exactly one thing well, until the question is reduced to a one-line answer. That is literally the Unix philosophy you talked about in this module's second lesson, put into practice with a single symbol.

Connection to the module: in the previous lesson you connected a command's output to a file — >, >>, tee. Now you are going to connect a command's output directly to another command's input, with nothing touching disk in between. It is the same idea of "moving a stream to another destination" you already know; the destination, this time, is a live process instead of a still file.

The | operator: an assembly line, not a warehouse in between

Think about a factory assembly line. One machine cuts the fabric, the next one sews it, the next one adds buttons. No half-finished piece gets stored in a warehouse between machines — it goes straight from one conveyor belt to the next, the moment it leaves one and before the other needs it. If you wanted to store every half-finished piece in a box, label it, and have the next worker go fetch it from the box, the process would end up the same, just much slower and with an extra step nobody asked for.

The | operator (the vertical bar, above the Enter key on most keyboards) does exactly that with commands: it takes the standard output (descriptor 1) of the command on the left and connects it directly to the standard input (descriptor 0) of the command on the right. Both commands start almost at the same time, as independent processes, and the operating system pushes the text from one to the other as it gets produced — with nothing written to disk, no temporary file you have to delete afterward.

command1 | command2

Compare it to the alternative with no pipe, using what you already know from the previous lesson:

command1 > temp.txt
command2 < temp.txt
rm temp.txt

Three lines, an intermediate file that exists only to be deleted a second later, and a name (temp.txt) you have to invent and remember to clean up. The pipe replaces the three lines with a single one, and leaves no stray file behind.

Worked example

You have the same access.log you used in the grep lesson from the previous module:

INFO  2026-07-18 09:12:03 User 42 logged in
ERROR 2026-07-18 09:14:51 Payment gateway timeout for order 1188
WARN  2026-07-18 09:15:02 Retry scheduled for order 1188
ERROR 2026-07-18 09:15:10 Payment gateway timeout for order 1188
INFO  2026-07-18 09:20:44 User 17 logged in
ERROR 2026-07-18 09:22:19 Database connection refused

Today's question is more specific than "show me the errors": you want to know how many of those errors are, specifically, from the payment gateway. You build the pipeline one link at a time, verifying each one before adding the next.

Step 1 — the first command, alone, verified.

grep ERROR access.log

What to expect:

ERROR 2026-07-18 09:14:51 Payment gateway timeout for order 1188
ERROR 2026-07-18 09:15:10 Payment gateway timeout for order 1188
ERROR 2026-07-18 09:22:19 Database connection refused

Three lines. You check they are the ones you expected — yes, they are the three errors — before continuing.

Step 2 — you add the second link and check again.

grep ERROR access.log | grep Payment

What to expect:

ERROR 2026-07-18 09:14:51 Payment gateway timeout for order 1188
ERROR 2026-07-18 09:15:10 Payment gateway timeout for order 1188

The three lines from grep ERROR went in, one by one, to the second grep's standard input, which kept only the ones that also contain "Payment." You went from three to two. Exactly what you were after: the "Database connection refused" line had nothing to do with payments, and it got left out.

Step 3 — you add the final link, the one that answers the question with a single number.

grep ERROR access.log | grep Payment | wc -l

What to expect:

2

Three commands, each doing exactly one thing: filtering by "ERROR," filtering by "Payment," counting lines. Neither knows anything about the others beyond "I receive text, I process it, I hand out text." That mutual ignorance is why you can combine them in any sensible order and with any number of links.

Build it piece by piece: verify each link before continuing

What you just did above was not a coincidence specific to this example — it is the method you want to use any time a pipeline has more than one link, especially if any of them can delete, move, or overwrite something:

1. Run the first command alone, with no | yet, and confirm its output is what you expected.
2. Add just one more link and look at the result again. If something came out different from what you thought, the problem is in that last link you just added — you do not have to suspect the whole chain.
3. Repeat until you reach the final command.
4. Only then, if the final result is going to feed something destructive or irreversible (an rm, an mv, a script that modifies production), run the full pipeline one more time to confirm it before letting it loose.

Writing the pipeline all at once, end to end, and only then running it, is the most common way to lose half an hour figuring out which of the four commands has the bug. Writing one link, running, looking, adding the next, is slower the first time and much faster overall.

Immediate combinations with what you already know

You do not need new tools for a pipeline to be useful to you starting today. With what you already know from previous modules, these three combinations solve real questions all the time:

Counting how many entries a command produced, without counting them by eye:

ls -l /etc | wc -l

What to expect: a number — the count of lines ls -l /etc printed — instead of the full listing scrolling across the screen.

Calmly reviewing the results of a recursive search, when there are too many to fit on a single screen:

grep -r "TODO" src/ | less

What to expect: instead of hundreds of matches flooding the terminal all at once, less shows them to you one screen at a time, and you can move forward with the space bar or search within that result with /.

Keeping only the first results of a search that can take a while and produce thousands of lines:

find / -name "*.py" 2> /dev/null | head

What to expect: the first ten lines (head's default) of the list of .py files across the whole system, instead of waiting for find to finish walking every directory before you see anything on screen at all.

In all three cases the same thing happened: a command you already knew produced too much output to look at directly, and a single-link pipeline turned it into something manageable.

The classic mistake: commands that do not read standard input

Not every command reads its standard input. Some — rm, cp, mv among the most common — only look at their command-line arguments (technically called argv): the list of names you typed after the command, separated by spaces. For those commands, whatever reaches them through a pipe is invisible; they do not even look at it.

Imagine a reports/ directory with temporary files a script left behind:

ls reports/

What to expect:

monthly-summary.tmp
weekly-summary.tmp
sales-summary.tmp

You try to delete them by chaining find with rm, as if rm were going to "receive" the names through the pipe:

find reports/ -name "*.tmp" | rm

What to expect: a usage message or a "missing operand," and no file deleted. The exact text varies by system — something like rm: missing operand on Linux (GNU coreutils), or a usage: rm [-f | -i] ... file ... on macOS (BSD rm) — but the result is identical on both: rm complains that it is missing a required argument, because, indeed, none was passed to it. The text find sent through the pipe reached rm's standard input, but rm never looks at its standard input to decide what to delete: it only looks at argv, and argv came in empty.

The solution is xargs. It reads whatever reaches it through standard input, line by line, and turns it into command-line arguments for the command that follows it:

find reports/ -name "*.tmp" | xargs rm

What to expect: nothing on screen — rm received monthly-summary.tmp weekly-summary.tmp sales-summary.tmp as real arguments, one by one, and all three files vanished with no complaint.

The strange name that breaks everything: spaces in a file name

Now a coworker leaves a file with a name that does not follow the team's convention:

find reports/ -name "*.tmp"

What to expect:

reports/old report.tmp

You repeat the same pipeline you just learned:

find reports/ -name "*.tmp" | xargs rm

What to expect:

rm: reports/old: No such file or directory
rm: report.tmp: No such file or directory

Nothing got deleted, and the two "files" rm tried to delete do not even exist. By default, xargs separates arguments by whitespace — just like you would separate words typing on the command line. The one line it received, reports/old report.tmp, got split into two: reports/old and report.tmp. Neither one is the real file.

The exact solution: find can end every result with a null byte (\0) instead of a newline, with -print0. And xargs -0 tells xargs to use that same null byte as the separator, instead of spaces or newlines:

find reports/ -name "*.tmp" -print0 | xargs -0 rm

What to expect: no output on screen — and reports/old report.tmp, the whole file name, space included, is gone.

Why the null byte and not any other separator? Because it is the only byte a file name cannot contain on a Unix system (the slash / cannot appear inside a name either, but you already use that one to separate directories). A space, a tab, and even a newline can all be a legitimate part of a real file name — unusual, but legal. The null byte, not so. That is why -print0 together with xargs -0 is the only combination that guarantees no name, no matter how strange, gets cut where it should not.

How errors behave inside a pipeline

The | operator connects exactly one stream: the standard output (descriptor 1) on the left with the standard input (descriptor 0) on the right. It never touches any command's standard error (descriptor 2) anywhere in the chain. This has a consequence that surprises people the first time they see it:

grep ERROR missing-file.log | wc -l

What to expect:

grep: missing-file.log: No such file or directory
0

Two things happened, and neither has anything to do with the other. grep could not open a file that does not exist, and it sent that error straight to the screen through its descriptor 2 — the pipe never touched it, because the pipe only moves descriptor 1. At the same time, since grep produced no line through its descriptor 1 (it had nothing to search), wc -l received a completely empty input, and correctly counted zero lines in it. The 0 you see does not mean "there were no errors in the log" — it means "the log, which does not even exist, sent no line to wc." They are two independent results, mixed on the same screen by the mere coincidence that both commands write there by default.

Every command in a pipeline runs as a separate process, almost simultaneously, and each one ends with its own internal success-or-failure result — that, and how to use it to chain decisions, is what this module's last lesson covers. For now, keep the central idea: a pipeline never hides or redirects errors on its own. If you want them to travel through the pipeline too, you need to ask for it explicitly with 2>&1, as you saw in the previous lesson.

Common mistakes

1. Thinking | also combines errors, same as 2>&1 (conceptual). What happens: you run command1 | command2 expecting that, if command1 fails, that error gets filtered or "absorbed" inside the pipeline, and instead you keep seeing it on screen, mixed in with whatever command2 prints. Why: | connects a single stream — descriptor 1 with descriptor 0 — never descriptor 2. Combining the errors requires asking for it separately, with 2>&1 before the |. How to spot it: you see error messages on the terminal that "should not be there" because you assumed they were already filtered or redirected. How to fix it: if you really want errors to enter the pipeline, write command1 2>&1 | command2.

2. Piping into a command that does not read standard input. What happens: you write find . -name "*.bak" | rm (or the same pattern with cp or mv) expecting the final command to "receive" the names through the pipe, and instead it fails immediately with a usage message (usage: ...) without touching a single file. Why: rm, cp, and mv take file names as command-line arguments, not from standard input; whatever reaches them through the pipe is invisible to them. How to spot it: the command ends almost instantly with a "usage" or "missing operand" message instead of complaining about a specific file. How to fix it: insert xargs between the two commands (find . -name "*.bak" | xargs rm), which does read standard input and turns it into arguments.

3. Using xargs with no -0 when file names can have spaces or newlines. What happens: find directory -name "*.tmp" | xargs rm fails with "No such file or directory" for files that do exist, or — worse — deletes something you never meant to touch because one of the cut-up fragments accidentally matched another real name. Why: without -0, both find and xargs use whitespace and newlines as separators; if a real name contains any of those characters, it gets split at the wrong spot and stops being a single argument. How to spot it: before piping into a destructive command, check whether any name in the directory has spaces (ls -l already shows you this, though it is easy not to notice). How to fix it: always use find ... -print0 | xargs -0 ... when the final command deletes, moves, or modifies files — the null byte is the only character no file name can contain, so it never cuts where it should not.

Exercises

1. You have an orders.log file with thousands of lines starting with INFO, WARN, or ERROR. You want to know how many error lines mention the word timeout. You have never written this pipeline before. Describe, step by step, how you would build and verify it — not just what the final command is.

2. You run grep ERROR sales.log | wc -l in a directory where sales.log does not exist. What exactly shows up on screen, and why does the number wc -l prints not mean what it looks like it means at first glance?

grep: sales.log: No such file or directory
0

3. This command fails and deletes nothing:

find backups/ -name "*.old" | rm

Why exactly does it fail, and how do you fix it?

find backups/ -name "*.old" | xargs rm

4. An archive/ directory has, among others, a file named final report.old (with a space in the name). You run find archive/ -name "*.old" | xargs rm and see this error:

rm: archive/final: No such file or directory
rm: report.old: No such file or directory

What went wrong, and what is the correct pipeline to delete that file no matter what strange characters its name has?

find archive/ -name "*.old" -print0 | xargs -0 rm

Summary and next step

You can now connect one command's output directly to the next one's input using |, with no intermediate file involved. You know how to build that pipeline piece by piece — verifying each link before adding the next — instead of writing the whole thing and only then finding out where it failed. You recognize when a chain's final command does not read standard input and needs xargs to receive the names as arguments, and you know why -print0 together with xargs -0 is the only safe combination when file names can have spaces or unusual characters. And you understood that a pipeline never hides errors on its own: every command in the chain keeps sending its error descriptor straight to the screen, unless you explicitly ask otherwise.

Everything you connected in this lesson was pieces you already knew from previous modules: ls, find, grep, wc, head, less. The pipeline itself is just plumbing — it connects, but does not transform anything on its own. What really multiplies its power are tools designed specifically to live inside a pipeline: taking raw text on one side and handing out sorted, counted, or translated text on the other. That is exactly what is next: sort, uniq, cut, tr, and sed, the text toolkit that turns a two-command pipeline into a real analysis instrument.

Before moving on you should be able to:

• explain in your own words exactly what the | operator connects, and what it does not connect (any command's error descriptor anywhere in the chain);
• build a pipeline of at least three commands, adding and verifying one link at a time, without writing the whole thing in one go;
• recognize when a command does not read standard input and needs xargs, and use -print0 together with xargs -0 when file names can have spaces or unusual characters.

Resources

• Pipelines — Bash Reference Manual — the official definition of the | operator in bash: how descriptors get connected and how a chain's commands run.
• Invoking xargs — GNU Findutils Manual — official xargs documentation: how it turns standard input into arguments and what options it controls.
• xargs(1) — Linux manual page — the full manual page for xargs, including the -0 option and its relationship to find -print0.
• find(1) — Linux manual page — full reference for find, with -print0's exact definition in the actions section.
• BashFAQ/020 — Greg's Wiki — why file names with spaces or newlines break naive pipelines, and why -print0 / -0 is the recommended solution over other alternatives.