2. The Unix philosophy: small tools that compose

Description

By the end of this lesson you will be able to explain why almost no serious terminal program tries to do everything, and you will have a concrete criterion for deciding whether a new problem gets solved by combining commands you already know instead of going out to install something different. This is not a historical curiosity: it is the mental filter that separates someone who memorized fifty commands from someone who actually understands what the terminal is for.

That criterion pays for itself the first time you apply it in real work. Someone who analyzes data and needs to know how many rows of a ten-million-line file have an empty value in a certain column does not open a spreadsheet or write a thirty-line script: they combine two or three commands that already exist on any Linux machine. An AI agent that runs commands in a terminal — like the ones that automate coding tasks today — does exactly the same thing: it does not have a special integration programmed for every tool, it has access to a terminal and to programs that read and write plain text. That is why it can combine find, grep, and wc with nobody ever having written code to connect those three programs to each other.

Connection to the module: the previous lesson showed you a pipeline working like a magic trick, without explaining why it was possible. This lesson gives you the underlying reason: it is not a syntax coincidence, it is a design decision from almost fifty years ago that made every program speak the same language. The next two lessons — the three streams and redirection — are going to show you the exact technical mechanism that makes that idea possible.

A hardware store, not an all-in-one factory

Think about a real hardware store, not the command with that name. In the plumbing section there are shutoff valves, elbows, T-joints, reducers, valves: dozens of small parts, each one solving exactly one problem (stopping the flow, changing direction, splitting into two branches). No manufacturer sells a single part that is simultaneously a valve, an elbow, and a reducer. And yet you can buy an elbow from one brand and a valve from a completely different one, and they fit with no adjustment, because both respect the same standard thread size. That standard measurement is what lets you build installations nobody designed as a whole: each manufacturer only needs to meet the standard, not know about everyone else's parts.

Unix was built on the same bet, and it is not a coincidence of vocabulary: the | symbol you already used in the previous lesson is literally called a "pipe" because it does the same thing as a plumbing elbow — it connects one piece to another without either one knowing anything about the other — and Unix's "standard thread size" is plain text: lines of readable characters, with no proprietary binary format in the way. As long as a program reads text on its input and writes text on its output, you can connect it to any other program that respects that same deal, no matter who wrote it, in what year, or in what language.

In 1978, in the foreword to the special issue Bell Labs' technical journal dedicated to Unix, Doug McIlroy — who headed the Bell Labs research center where Unix was born, and who had already proposed the idea of pipes back in 1964, the same idea Ken Thompson would end up implementing in 1973 — summed up the idea in three sentences still quoted in their original language, word for word:

«Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.»

Translated: write programs that do one thing and do it well; write programs that work together; write programs that handle text streams, because that is a universal interface. That is three decisions, not one, and none of them works alone. Small, focused programs (grep searches, it does not count or sort). Programs that combine instead of competing to do everything. And plain text as the common language that makes the combining possible: if grep did its job perfectly but wrote its own binary format, you would not be able to connect it to anything you already know.

Worked example

You are going to reproduce this in your own terminal; no special file is needed, just what you already learned in module 2.

# create a practice folder with empty files: we only care about the names
mkdir -p ~/unix-philosophy-practice
cd ~/unix-philosophy-practice
touch notes.txt draft.md README.md report.md ideas.txt config.yaml

Now let us answer a concrete question: how many Markdown (.md) files are in this folder?

find knows how to locate files by pattern, but does not know how to count:

find . -name "*.md"

What to expect:

./README.md
./draft.md
./report.md

You have the list, not the number. And wc — module 2's tool that counts lines, words, and characters — does not know how to search for anything on its own: if you hand it a folder name, it has no way to decide which files you care about. Neither one, on its own, answers the full question. Connect them:

find . -name "*.md" | wc -l

What to expect:

3

Nothing changed on disk and you did not install any new tool. find keeps doing exactly one thing (locating) and wc keeps doing exactly one thing (counting); the composite question — "how many files of this type are there?" — gets solved by the combination, not by a third program someone had to write custom-built for this specific case. This is the philosophy working in practice, not an abstract idea.

The criterion before the tool

The most expensive reflex in the terminal is not mistyping a command: it is not asking yourself, before installing or programming something, whether what you already have is enough. The criterion is simple and it is worth applying in this order:

1. Isolate the atomic question. Not "I need a log analyzer," but "how many lines meet this exact condition?"
2. Check which commands you already know that solve a piece of it. There is almost always one that filters (grep, find) and one that counts or transforms (wc, and later sort, cut).
3. Connect those pieces before going out to look for something new. If the combination exists, you do not need anything else.

Apply it to a real case. A teammate tells you: "I need a tool that tells me how many lines of this log mention WARNING but do not mention ignored, so I know how many real warnings are still unresolved." Before searching "log analyzer" online, apply the criterion: the atomic question is a conditional count; grep already filters by pattern and already knows how to invert the search with -v (module 2); wc -l already counts lines. Chained together:

grep "WARNING" server.log | grep -v "ignored" | wc -l

Two filters and a count, no new program. Installing a dedicated tool is only justified once the combination stops being enough — for example, when the condition is no longer about plain text, which is exactly the limit coming up next.

Where plain text falls short

The philosophy does not say "everything is text and that is enough"; it says plain text is the universal interface for data that reads line by line. That has a real limit, and it is worth stating it with the same honesty as the advantage.

Nested-structure data. A JSON file is text, but it is not line-by-line text: the same object can be on a single very long line or spread across fifty, and its fields' order is not guaranteed. grep searches for character patterns, not concepts; if you search for the word email in a JSON file, it can find either the value you need or the "email": key somewhere you do not care about, or find nothing at all if the file comes compressed onto a single long line. That is what jq exists for: a program that reads structure (objects, lists, keys) instead of characters, and lets you ask "which users have "active": true?" on its own terms. It is worth noting something: jq does not break the philosophy, it extends it — it still reads text on its input, still writes text on its output, still combines with pipes just like grep — it just understands a richer structure than a plain line.

Delimiters that lie. A CSV file looks like perfect plain text for cut -d, -f2, until a field carries a comma inside its own quotes ("Doe, John",42,Lima). cut knows nothing about quotes: it counts commas blindly and hands you the wrong column, with no error to warn you. There, the text is still text, but the delimiter stopped being reliable; you need a tool that really understands the CSV format (csvkit, for example), not a cut by character position.

Binary data. An image, a compiled executable, or a SQLite database file have no "lines" in any useful sense. Running them through grep or cat does not produce a polite error: it produces unreadable characters on your screen or, in the worst case, a terminal you need to restart. Those formats have their own specialized tools (an image editor, a debugger, a SQL client), and forcing them into the text pipeline is not faithfulness to the philosophy: it is ignoring the limit the philosophy itself declares.

Common mistakes

Believing "following the Unix philosophy" means only using commands from the 1970s. What happens: someone rejects modern tools like ripgrep or jq arguing "they are not real Unix," and stays stuck with slower or less convenient versions of the same idea. Why it happens: the philosophy describes a behavior — one thing, plain text, composition — not a birth date. ripgrep searches for patterns, reads and writes text, combines in pipelines exactly like grep; it just also respects .gitignore and is faster. It is as faithful to the philosophy as the original, or more. How to spot it: if your argument for avoiding a tool is "it is too new" and not "it does more than one thing" or "it does not speak plain text," the argument has nothing to do with the philosophy and everything to do with habit. How to fix it: evaluate any tool, old or new, with McIlroy's three questions — does it do one thing? does it speak text? does it combine? — not with its release year.

Writing a script from scratch, or installing a package, for a problem two connected commands already solve. What happens: you spend twenty minutes writing and debugging a thirty-line script — or add a new dependency to a project — for something find ... | wc -l or grep ... | grep -v ... solves in a single line. Why it happens: you are missing the habit of applying the previous section's criterion before typing the first line of code; the default reflex is "program" instead of "combine." How to spot it: if your solution has more lines than the question it answered, or if you added a dependency for just a one-off count or filter, there was probably a shorter pipeline waiting. How to fix it: before opening an editor, write in one line what the atomic question you need to answer is, and check whether two commands you already know solve it chained together.

Using cut or grep on a format that is not plain line-by-line text, and trusting the result without checking it. What happens: cut -d, -f2 on a CSV with a quoted field containing a comma hands you the wrong piece of a field, and the command does not fail: it just lies silently. Why it happens: cut counts the delimiter character literally, with no knowledge of quotes or the format's structure; it does exactly what it was asked, which was not what was needed. How to spot it: a column's values do not match what you expected, or they show up shifted exactly on the rows that have the delimiter embedded inside a field. How to fix it: if the file is real CSV, use a tool that understands quotes and structure (csvkit or another one); if it is JSON, use jq. Reserve cut for columns separated by a simple, exception-free delimiter, which is exactly the case it was designed for.

Exercises

Exercise 1

Reuse the ~/unix-philosophy-practice folder from the worked example (or create a new one with mkdir and touch) and add these files: access.log, error.log, debug.log, main.py, utils.py. Without using any tool that "counts files by type," answer each question with a single command: how many .log files are there? How many files are NOT .py?

find . -type f -name "*.log" | wc -l
find . -type f ! -name "*.py" | wc -l

Exercise 2

A teammate needs to know how many configuration files (.yaml or .yml) are in a project, not counting the ones inside the tests folder. Apply this lesson's criterion: what combination of already-known commands answers this without installing anything?

find . \( -name "*.yaml" -o -name "*.yml" \) | grep -v "/tests/" | wc -l

Exercise 3

You have a JSON configuration file with a list of users, each with fields like email and active. You want to extract just the email addresses of the active users. Why are grep and cut not the right tool here, even though the file is "text"? What would you use instead?

Exercise 4

Which of these two statements is more faithful to the Unix philosophy: (a) "never install a tool that has not existed since the 1970s," or (b) "before installing something new, ask yourself whether connecting what you already have solves the problem"? Justify your answer with an example of a modern tool that does respect the philosophy.

Summary and next step

You now know why the terminal behaves like a language and not like a collection of isolated applications: every program does one thing, all of them speak plain text, and the real power lives in the combination. You also have a concrete criterion — isolate the atomic question, look for which piece each already-known command solves, combine before installing — and you know where that approach stops being enough: nested data like JSON, delimiters that lie, and binaries with no lines.

What you have not seen yet is the exact technical mechanism that makes connecting two programs possible: what a program "writes" when it writes text, and why an error still shows up on your screen even though you redirected the output to a file. That is exactly the next lesson: the three streams every process has from the moment it is born.

Before moving on you should be able to explain, in your own words and without looking at this lesson, why grep and wc can be combined even though different people wrote them at different times, and apply this lesson's criterion to a new problem before installing anything.

Resources

• UNIX Time-Sharing System: Foreword — the 1978 special issue of the Bell System Technical Journal where Doug McIlroy wrote the original formulation of the philosophy quoted in this lesson.
• The Art of Unix Programming — chapter 1 — Eric S. Raymond's full treatment of Unix's design principles, with historical and contemporary examples.
• Program Design in the Unix Environment — the article by Rob Pike and Brian Kernighan (1983) showing, with real code examples, how a program designed to combine with others gets built.
• Official jq manual — full reference for the tool that extends the philosophy to structured JSON data.
• ripgrep on GitHub — the project's README explains in detail why a modern tool still respects (and in some ways improves on) the 1978 principles.