6. The text toolkit: sort, uniq, cut, tr, and sed

Description

By the end of this lesson you will be able to sort lines of text by any criterion, count how many times each distinct value repeats, keep only the columns you care about from a file with irregular formatting, translate or delete specific characters, and make text substitutions — all chained into a single pipeline that answers, in seconds, questions about a file with tens of thousands of lines.

This is the skill behind questions that come up every week in real work: which IP address hit the server the most times this morning? How many distinct customers show up in this sales export? How many lines of this error log correspond to each status code? Without this kit, those questions get answered by opening the file in a text editor and counting by eye — something that works with a hundred lines and becomes impossible with a hundred thousand.

Connection to the module: in the previous lesson you learned the | operator to connect one command's output to the next one's input, and you tried it with generic pieces (grep, wc, head, xargs). Now you are going to meet the specialized pieces that make those pipelines actually analyze text instead of just filtering or counting it raw.

A workbench for lines of text

None of these five commands knows what a "table" is in the spreadsheet sense. All they see is text: lines separated by newlines and, inside each line, loose characters. But if you tell them where each "column" starts and ends — almost always by marking it with a character that repeats, like a comma or a space — that plain text starts behaving just like a spreadsheet: it can be sorted by a column, you can count how many rows repeat the same value, you can keep two columns and discard the rest.

sort sorts whole lines. uniq collapses and counts the ones that are identical. cut trims columns. tr translates or deletes characters, one at a time. sed searches for a text pattern and replaces it. Each one does exactly one thing — the same Unix philosophy you already saw — and the real value shows up when you connect them with | in the right sequence.

Worked example: the universal counting pattern

Create this test file, a short version of an access log:

cat > access.log << 'EOF'
203.0.113.5 - - [21/Jul/2026:10:02:13 +0000] "GET /index.html HTTP/1.1" 200
198.51.100.23 - - [21/Jul/2026:10:02:15 +0000] "GET /about.html HTTP/1.1" 200
203.0.113.5 - - [21/Jul/2026:10:02:18 +0000] "GET /contact.html HTTP/1.1" 200
203.0.113.5 - - [21/Jul/2026:10:02:20 +0000] "GET /index.html HTTP/1.1" 200
198.51.100.23 - - [21/Jul/2026:10:02:22 +0000] "GET /login.html HTTP/1.1" 404
203.0.113.5 - - [21/Jul/2026:10:02:25 +0000] "GET /index.html HTTP/1.1" 200
192.0.2.44 - - [21/Jul/2026:10:02:30 +0000] "GET /pricing.html HTTP/1.1" 200
198.51.100.23 - - [21/Jul/2026:10:02:33 +0000] "GET /about.html HTTP/1.1" 200
EOF

The question: which IP address made the most requests? Building the pipeline piece by piece, as you already practiced in the previous lesson:

cut -d' ' -f1 access.log

What to expect: the first column of every line, one IP per row, in the same order they show up in the file — still not sorted or counted.

cut -d' ' -f1 access.log | sort

What to expect: the same eight IPs, now grouped — every repetition of the same IP ends up next to each other. This is exactly what the next link needs.

cut -d' ' -f1 access.log | sort | uniq -c

What to expect:

      1 192.0.2.44
      3 198.51.100.23
      4 203.0.113.5

Every unique line, preceded by how many times it showed up. But it is sorted alphabetically by IP, not by frequency — one last link is missing for that:

cut -d' ' -f1 access.log | sort | uniq -c | sort -rn | head -3

What to expect:

      4 203.0.113.5
      3 198.51.100.23
      1 192.0.2.44

There is the answer: 203.0.113.5 made more requests than any other. On a real file with a hundred thousand lines, head -3 is what saves you from having to look at a hundred thousand results.

The pattern sort | uniq -c | sort -rn | head is probably the pipeline you are going to rewrite the most in your career: it works for IPs, for error codes, for users, for products — for any file where the question is "which values repeat the most?"

sort: sorting with judgment (-n, -r, -u, -k)

By default, sort sorts line by line, character by character, in alphabetical order (technically, by byte value). That is enough for names, but it fails in a very specific way with numbers: for sort with no -n, the string "10" comes before "9", because it compares the first character (1 versus 9) and 1 is "smaller" alphabetically, no matter that 10 is bigger than 9 as a number.

cat > scores.txt << 'EOF'
9
10
2
EOF

sort scores.txt

What to expect:

10
2
9

That order is correct for sort, but it is probably not the one you want. The -n option tells it to compare numeric values, not characters:

sort -n scores.txt

What to expect:

2
9
10

The other three options you are going to use often:

• -r reverses the order (largest to smallest, or Z to A).
• -u removes duplicate lines while sorting — it is the equivalent of sort | uniq, but in a single step, when all you need is the list of distinct values, without the count.
• -k N sorts by column N instead of the whole line. It needs -t to tell it what the column separator is if it is not spaces.

cat > products.csv << 'EOF'
mouse,electronics,25
desk,furniture,120
keyboard,electronics,45
chair,furniture,80
monitor,electronics,150
EOF

sort -t',' -k3 -n products.csv

What to expect:

mouse,electronics,25
keyboard,electronics,45
chair,furniture,80
desk,furniture,120
monitor,electronics,150

Sorted by price, smallest to largest. Here we use -k3 (not -k3,3) because the price is the last column: there is nothing after it that could sneak into the comparison. But -k N with no second number extends the comparison key from field N to the end of the line, not just that field — so if you wanted to sort by category (column 2) ignoring price as a tiebreaker, you would need -k2,2, to constrain the key exactly to that column.

uniq and uniq -c: why they require an already-sorted input

uniq removes consecutive repeated lines and counts them with -c. The key word is consecutive: uniq does not compare every line against every other line in the file, only against the line immediately before it. If two identical lines are separated by different ones, uniq does not detect them as duplicates.

cat > visits.txt << 'EOF'
/home
/about
/home
EOF

uniq -c visits.txt

What to expect:

      1 /home
      1 /about
      1 /home

Three lines, three counts of "1" — uniq did not notice /home shows up twice, because between the two occurrences there was a different line. That is why the pattern is always sort first:

sort visits.txt | uniq -c

What to expect:

      2 /home
      1 /about

sort grouped the two /home lines next to each other; only then could uniq -c count them together. This dependency — uniq needs identical lines to be adjacent, and sort is what makes them adjacent — is why you are almost never going to write uniq without a sort before it in the same pipeline.

cut -d -f: keeping columns (and their limits)

cut extracts columns from every line. -d sets the delimiter character and -f which columns (fields) you want, counting from 1.

cut -d',' -f1 products.csv

What to expect:

mouse
desk
keyboard
chair
monitor

You can ask for several columns separated by a comma (-f1,3) or a range (-f1-2).

cut's real limit shows up with irregular separators — when the same field can be separated by a variable amount of the delimiter, as happens with ls -l's output, which aligns columns with a number of spaces that changes based on each value's width:

ls -l /etc | head -3

What to expect (this is an illustrative example: on your own machine the total, the files, the sizes, and the spacing are going to be different — what matters is the shape, not these exact values):

total 1288
drwxr-xr-x   3 root  wheel    96 Mar  3 09:15 apache2
-rw-r--r--   1 root  wheel  1928 Mar  3 09:15 afpovertcp.cfg

If you try cut -d' ' -f2 expecting "the number of links," the result is an empty string or a wrong value on some lines: cut treats every individual space as a new delimiter, so two or three spaces in a row — the ones ls uses to align columns — generate several empty fields before reaching the real value, and how many empty fields show up depends on how much padding that particular line has. The field number that holds the data you are looking for ends up being different line by line.

This is different from how sort separates columns by default (with no -t), which does treat a run of spaces as a single separator — that is exactly why cut needs a consistent, single-character delimiter to work well, and why the next section shows how to fix this specific case before handing it to cut.

tr: translating and deleting characters

tr does not work with a file as an argument — it only reads from standard input, same as you saw with < in the redirection lesson — and it translates or deletes characters, one at a time, not whole text patterns (that is sed's job, below).

Translating one set of characters into another:

echo "Deploy Started" | tr 'a-z' 'A-Z'

What to expect:

DEPLOY STARTED

Deleting characters with -d — useful, for example, to clean up the carriage returns (\r) that files created on Windows leave behind, and that on macOS or Linux show up as a ^M stuck to the end of every line:

tr -d '\r' < windows_file.txt > clean_file.txt

Squeeze with -s collapses consecutive repetitions of a character into a single occurrence — and this is exactly what solves the ls -l problem you saw in the previous section:

ls -l /etc | tr -s ' ' | cut -d' ' -f2 | head -3

What to expect (continuing with the same illustrative /etc example from above — on your system these three numbers are going to be different, but every line is going to consistently give you the link count, with no empty fields sneaking in between):

1288
3
1

tr -s ' ' collapsed every run of repeated spaces into one before cut ever saw the line, so now every individual space really does correspond to a real column boundary, and cut -d' ' -f2 gets the link count consistently on every line.

sed 's/old/new/g': text substitutions

sed searches for a pattern and replaces it. The most common form is s/pattern/replacement/, where s means "substitute" and the slashes separate the three parts.

cat > pets.txt << 'EOF'
cat cat dog
dog cat cat
EOF

sed 's/cat/dog/' pets.txt

What to expect:

dog cat dog
dog dog cat

Without the g flag, sed replaces only the first match on each line — that is why on the first line only the first cat changed, and on the second only the first of the two. With g ("global"), it replaces every match on every line:

sed 's/cat/dog/g' pets.txt

What to expect:

dog dog dog
dog dog dog

By default, sed prints the result on screen and does not touch the original file — pets.txt still has cat if you check it after these two commands.

The warning about -i. To edit the file directly, sed has the -i flag — but it behaves differently on Linux (GNU sed) and on macOS (BSD sed, the one the system ships with out of the box):

# On Linux (GNU sed): works as-is, no backup
sed -i 's/cat/dog/g' pets.txt

# On macOS (BSD sed), the same command fails
sed -i 's/cat/dog/g' pets.txt

What to expect on macOS: a syntax error (something like extra characters at the end of s command), not the substitution you were after. The reason: BSD sed requires -i to always receive an argument — the backup file's suffix — even when you do not want a backup. Since you did not give it one explicitly, it takes 's/cat/dog/g' as if it were that suffix and pets.txt as if it were the script to run, and neither interpretation is the one you wanted.

The correct form on macOS, passing an empty string as the suffix:

sed -i '' 's/cat/dog/g' pets.txt

And the form that works the same on both platforms, if you need a portable script — it leaves a backup with the .bak extension:

sed -i.bak 's/cat/dog/g' pets.txt

This is the only real incompatibility you are going to run into often among this lesson's five tools: sort, uniq, cut, and tr, with the options you saw here, behave the same on macOS and on Linux.

An honest limit: where this kit ends

awk exists, and for working by columns it is more powerful than cut, tr, and sed combined — it can do arithmetic, conditionals, and format output in a single line. It is deliberately left out of this guide: this lesson's kit solves the vast majority of everyday tasks, and awk is, on its own, enough material for a whole other topic. When you run into a problem these five tools do not solve comfortably, that is the signal it is time to learn it.

Common mistakes

1. Thinking uniq deduplicates the whole file, regardless of order (conceptual). What happens: someone runs uniq -c file.txt expecting the real count of each distinct value, and the result shows almost everything with a count of 1, as if nothing repeated. Why: uniq only compares each line against the one immediately before it — if the repeats are not adjacent, it does not see them as duplicates. How to spot it: compare wc -l file.txt against sort file.txt | uniq | wc -l; if the second number is much smaller, there were duplicates uniq alone was not seeing. How to fix it: always sort file.txt | uniq -c, never uniq -c file.txt on its own, unless you already know for certain the file comes pre-sorted.

2. Using sed -i the same way on macOS as on Linux. What happens: a script that works perfectly on a Linux server fails on a teammate's macOS laptop, with a syntax error that has nothing to do with the substitution itself. Why: BSD sed (macOS) requires -i to receive an explicit argument — the backup suffix — while GNU sed (Linux) treats it as optional. How to spot it: the error message mentions something like extra characters or treats your input file as if it were a sed script. How to fix it: use sed -i '' 's/.../.../g' file on macOS, or adopt the portable form sed -i.bak if the script has to run on both platforms unchanged.

3. Using cut -d' ' on text with irregular spacing. What happens: you extract what you think is column N and the result is inconsistent — empty on some lines, with the wrong value on others. Why: cut counts every individual occurrence of the delimiter as a new column boundary; if the amount of padding spaces changes from line to line (as in ls -l), the field number holding your data changes along with it. How to spot it: compare cut's result against what you see at a glance on a couple of different lines in the file — if it does not match consistently, the delimiter is not as simple as it looks. How to fix it: collapse repeated spaces first with tr -s ' ' before handing the result to cut.

Exercises

1. You have this file with visited pages, in the order they were visited (not sorted):

cat > pages.txt << 'EOF'
/home
/about
/home
/pricing
/about
/home
/contact
/about
EOF

Write a single pipeline that tells you, from most to least visited, how many times each page was visited.

sort pages.txt | uniq -c | sort -rn

      3 /home
      3 /about
      1 /pricing
      1 /contact

2. Using the lesson's products.csv (mouse,electronics,25 / desk,furniture,120 / keyboard,electronics,45 / chair,furniture,80 / monitor,electronics,150), get the list of prices sorted smallest to largest, without the other two columns.

cut -d',' -f3 products.csv | sort -n

25
45
80
120
150

3. You receive notes.txt, a file exported from Windows, and you notice every line ends with a visible ^M character when you open it with cat -A notes.txt. Clean the file into a new copy called notes_clean.txt without using a text editor.

tr -d '\r' < notes.txt > notes_clean.txt

4. You have app.conf with several lines containing the word TODO, and you need to change them all to DONE, editing the file directly but keeping a backup, with a command that works the same on Linux and on macOS.

sed -i.bak 's/TODO/DONE/g' app.conf

Summary and next step

You now have the full kit to turn plain text into answers: sorting with sort (and its variants -n, -r, -u, -k), counting with uniq -c on an already-sorted input, trimming columns with cut -d -f while knowing where it breaks with irregular spacing, translating or deleting characters with tr, and substituting text with sed 's/.../.../g' without accidentally clobbering the original file on macOS. And you have the pattern that ties them all together: sort | uniq -c | sort -rn | head.

Up to now, every pipeline you built silently assumed every command along the way worked. That is not always true: a grep might find nothing, a sed -i might fail for the reason you just saw, a file might not exist. What comes next is how to know — for certain, not by guessing — whether a command succeeded or failed, and how to chain commands that react to that result.

Before moving on you should be able to:

• explain, without looking at this lesson, why uniq -c file.txt with no sort before it usually gives incorrect counts;
• write from memory the pipeline sort | uniq -c | sort -rn | head to find the most frequent values in any text file;
• recognize, on seeing a sed -i error on macOS, that the problem is the flag's syntax and not the substitution itself.

Resources

• sort invocation — GNU Coreutils Manual — official reference for all of sort's options, including the full -k syntax with a key's start and end.
• uniq invocation — GNU Coreutils Manual — official uniq documentation, with the rest of the options (-d, -u) this lesson did not cover.
• cut(1) — Linux manual page — the full manual page for cut, with the exact detail of how it interprets the delimiter.
• tr(1) — Linux manual page — tr's manual page, with the full syntax for character ranges and the -d and -s flags.
• sed, a stream editor — GNU sed Manual — official GNU sed manual, with the s/// command and all its flags in detail.