Module 4: Permissions, Processes, and Environment

8. Project: the clinic of a broken environment

Description

By the end of this project you will be able to diagnose and repair, with evidence and not superstition, three failures any developer runs into during their first week on a real team: a command that exists on disk but that your shell cannot find, a script that refuses to run alongside a directory that refuses to let you in, and a port that someone — probably you, twenty minutes ago — left held by a background process. The deliverable is not just "it works now": it is a log of the three incidents, in the same format a real operations team uses — symptom, hypothesis, diagnostic command, root cause, fix, and verification.

This is exactly the first-day ritual on any team. They hand you a laptop, you clone a repository, and something — almost always something small — does not work on the first try. An experienced coworker does not reinstall the operating system or copy a sudo chmod 777 from a forum they found at two in the morning: they open a terminal, isolate the failing variable, and fix it in minutes because they know exactly what to ask the system. That difference — between guessing and diagnosing — is what separates someone the team can put in front of a production server unsupervised from someone who cannot yet.

Connection to the module: every previous lesson in this module gave you a loose piece — users and groups, permissions and chmod, when sudo is the right tool and when it is a symptom of a different error, processes and signals, the PATH, and the rc files that make a change persistent. Today you learn no new piece: you combine all six against three real, reproducible failures you are going to trigger yourself and then repair with the same rigor you would use on a server that is not yours.

Before touching anything: an incident's runbook

A doctor facing a symptom — fever, pain — does not prescribe the first treatment that comes to mind. They separate the symptom (what the patient reports) from the hypothesis (their initial suspicion), order a test that confirms or rules out that hypothesis, and only then treat the real cause, not the symptom. An engineer facing a Permission denied or a command not found has exactly the same work ahead of them, and the exact opposite temptation: try sudo, then chmod 777, then restart the laptop, until something "works" with no understanding ever gained of what was broken.

The professional alternative is a runbook: a fixed template you fill out for every incident, in this order, with no steps skipped.

Field	What it answers
Symptom	What you literally see on screen, word for word.
Hypothesis	Your best guess at the cause, before confirming it with a command.
Diagnosis	The exact command you ran to confirm or rule out the hypothesis, and its real output.
Root cause	What is really going on — not the symptom, the reason behind it.
Fix	The command that solves the root cause, not the one that silences the symptom.
Verification	How you confirm, with a command, that it got resolved.
Persistence	Whether the fix survives closing the terminal, or whether it needs to be saved in your rc file so you do not repeat it every session.

You are going to fill out this template three times — once per incident — and at the end you are going to have the project's complete log.

Incident 1: the phantom command

Trigger the failure by installing a tool of your own in a folder you control, something you are going to repeat many times in your real career:

mkdir -p ~/bin
cat > ~/bin/greet <<'EOF'
#!/usr/bin/env bash
# Greets using the system's real user
echo "Hello, $(whoami). All set."
EOF
chmod +x ~/bin/greet

Worked example

Symptom. You type the name of the command you just installed and the shell tells you it does not exist:

$ greet
zsh: command not found: greet

Hypothesis. Your first, reasonable, suspicion is that the file did not get created correctly or that it is missing execute permission.

Diagnosis.

$ ls -l ~/bin/greet
-rwxr-xr-x 1 ana staff 96 Jul 21 09:40 /Users/ana/bin/greet

The three xs (one per class: owner, group, others) rule out the hypothesis: the file exists and does have execute permission. The problem is something else. You confirm with the command that answers exactly this question:

$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

$ which greet
$ echo $?
1

which printed nothing, and exit code 1 confirms no folder in $PATH contains that executable.

Root cause. The file exists and has execute permission, but it lives in ~/bin, a folder that shows up nowhere in $PATH. The shell resolves a slash-free command by searching only those folders, in that exact order — it does not search the whole disk, nor does it "guess" where something you installed might be.

Fix (temporary, lasts only this session).

$ export PATH="$HOME/bin:$PATH"
$ greet
Hello, ana. All set.

Persistence. A PATH change made by hand with export lives only in this shell's memory — it disappears the moment you close the terminal. Just like you learned in the previous lesson, you leave it written in the correct rc file:

echo 'export PATH="$HOME/bin:$PATH"' >> ~/.zshrc
source ~/.zshrc

Verification. You open a new tab (or simply confirm after the source) and run which greet — now it does return /Users/ana/bin/greet, with nothing exported by hand.

Incident 2: the missing permission (script and directory)

This incident has a trap: two completely different root causes produce the same superficial message, Permission denied. Trigger it with a deployment script stored in a folder you accidentally stripped the enter permission from:

mkdir -p ~/deploy_tools
cat > ~/deploy_tools/deploy.sh <<'EOF'
#!/usr/bin/env bash
# Simulates a deployment step
echo "Deploying the application..."
EOF
chmod 600 ~/deploy_tools

That last line simulates a real, frequent mistake: someone, trying to "secure" the deployment folder, stripped the directory's execute bit without realizing the consequence.

Worked example

Symptom (first failure).

$ cd ~/deploy_tools
cd: permission denied: /Users/ana/deploy_tools

Hypothesis. That the folder does not exist, or got corrupted.

Diagnosis.

$ ls -ld ~/deploy_tools
drw------- 2 ana staff 64 Jul 21 09:52 /Users/ana/deploy_tools

Root cause. This rules out the hypothesis: the folder exists just fine. What is missing is the owner's x bit. On a directory, x does not mean "execute" like on a file — it means traverse: without it you cannot cd in, or access the metadata of what is inside, even with read permission (r) on the folder itself.

Fix.

$ chmod u+x ~/deploy_tools
$ ls -ld ~/deploy_tools
drwx------ 2 ana staff 64 Jul 21 09:52 /Users/ana/deploy_tools
$ cd ~/deploy_tools

With the directory open, you try running the script and the second failure shows up, with an almost identical message to the previous one:

$ ./deploy.sh
zsh: permission denied: ./deploy.sh

Diagnosis.

$ ls -l deploy.sh
-rw-r--r-- 1 ana staff 52 Jul 21 09:51 deploy.sh

Root cause (different from the previous one). Now it is the file, not the directory, that is missing its own x bit. The one on deploy_tools you just fixed only controlled whether you could enter the folder — it never controlled whether the file inside was executable. They are two completely independent permissions, even though the error message looks the same.

Fix.

$ chmod u+x deploy.sh
$ ./deploy.sh
Deploying the application...

Verification. ls -l deploy.sh now shows -rwxr--r--, with the owner's x present, and the script runs with no error.

Persistence. This incident is different from the previous one: the permission bits are already persistent on their own — they live on disk, not in the shell's memory — so there is nothing to add to an rc file. What is worth writing down is the cause in the team's log, so nobody repeats the mistake of "securing" a folder by stripping its traverse bit. And if deploy.sh lives in a git repository, it is worth confirming the execute permission got recorded in the repository itself with git ls-files -s deploy.sh (it should show mode 100755, not 100644) — that way, when a coworker clones the project, the script arrives already executable, with no need for them to discover this same incident on their own.

Incident 3: the busy port

Trigger the failure by launching a test server in the background, something you will do constantly while developing:

$ python3 -m http.server 8000 &
[1] 41213
Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ...

Twenty minutes later, that server already forgotten, you try to spin up another one on the same port:

$ python3 -m http.server 8000
Traceback (most recent call last):
  ...
OSError: [Errno 48] Address already in use

(On Linux the same error shows up as [Errno 98] instead of [Errno 48] — the errno number changes between macOS/BSD and Linux, but the message and the cause are identical.)

Worked example

Symptom. The second server does not start; Python ends with an OSError pointing out the address is already in use.

Hypothesis. An earlier process is still alive and still listening on port 8000.

Diagnosis.

$ lsof -i :8000
COMMAND   PID USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
Python  41213  ana    3u  IPv4 0x5f3a9c8b2b1e4a2f      0t0  TCP *:8000 (LISTEN)

Root cause. PID 41213 is the same number the shell printed as job [1] when you launched the first server with &. That process never got closed — it stayed alive in the background, invisible the moment you stopped watching it, holding the port.

Fix.

$ kill 41213
[1]  + terminated  python3 -m http.server 8000
$ lsof -i :8000
$

lsof -i :8000 no longer prints anything: the port is free. kill, with no signal number, sends SIGTERM (15) by default: it asks the process to close in an orderly way, giving it the chance to properly release its resources. If a process does not respond to SIGTERM (for example, because it got stuck or is ignoring the signal), the last resort is kill -9 41213, which sends SIGKILL: the kernel ends the process immediately, with no chance to close in order. That is why -9 is the last step, not the first — you can leave temporary files or connections half-closed if you use it from the start.

Verification. lsof -i :8000 with no output confirms nothing is still listening there; the second python3 -m http.server 8000 now starts with no error.

Persistence. Here there is no value to save in the rc file, like in Incident 1 — what is worth persisting is the tool that avoids manually repeating the full diagnosis every time this happens to you again:

# ~/.zshrc (or ~/.bashrc)
killport() {
  local pid
  pid=$(lsof -ti ":$1")
  if [ -n "$pid" ]; then
    kill "$pid"
    echo "Ended process $pid which was holding port $1."
  else
    echo "Nothing listening on port $1."
  fi
}

$ source ~/.zshrc
$ killport 8000
Nothing listening on port 8000.

The next time a port ends up busy, killport 8000 replaces the three commands you just ran by hand.

The project's final log

The three incidents, summarized in the format you would hand your team:

#	Symptom	Root cause	Fix	Persistence
1	`command not found: greet`	`~/bin` is not in `$PATH`	`export PATH="$HOME/bin:$PATH"`	Line added to `~/.zshrc`
2	`permission denied` entering and running	Missing `x` on the directory and, separately, on the file	`chmod u+x ~/deploy_tools` and `chmod u+x deploy.sh`	Already persistent on disk; documented for the team
3	`Address already in use` on port 8000	Earlier background process never closed	`kill 41213` (or `kill -9` if unresponsive)	`killport` function added to `~/.zshrc`

Common mistakes

"sudo fixes everything" (conceptual). What happens: faced with Incident 1's command not found: greet, the instinctive reaction is to type sudo greet, reasoning "sudo" solves any permissions problem. The error persists, identical. Why it happens: sudo elevates the user a command runs as — it does not change at all how the shell searches for that command in $PATH. In fact, many sudo configurations use their own secure_path, different from yours, which can hide the executable even further. The problem was never about privileges; it was about where the shell searches. How to spot it: if sudo command still returns command not found, you already know the problem is not about superuser permissions — it never was. How to fix it: before escalating privileges, diagnose with which, type, or echo $PATH whether the problem is path resolution; reserve sudo for when the diagnosis confirms it really is a missing system permission.

"If chmod does not fix the problem, I'll try 777" (conceptual). What happens: faced with Incident 2, instead of identifying exactly which bit is missing (x on the directory, x on the file), the reflex is running chmod -R 777 over the whole folder "so it stops giving problems." And it really does stop giving problems — that is exactly the risk. Why it happens: 777 grants read, write, and execute permission to any user on the system, not just the owner, violating the principle of least privilege you saw at the start of the module. On a single-user personal laptop the immediate cost looks like zero; on a shared server, any other account on the system can now read, modify, or replace that script. How to spot it: ls -l showing rwxrwxrwx on a file or folder that has no reason to be accessible by everyone. How to fix it: identify the exact missing bit with ls -l/ls -ld, and apply the minimum permission needed (chmod u+x, or the octal equivalent) instead of the maximum available — the goal is never "make it stop failing," it is "give it exactly the access it needs."

Using kill -9 as the first move (procedural). What happens: in Incident 3, instead of a plain kill, kill -9 41213 gets typed directly because "it's safer, this way it definitely dies." Why it happens: "more aggressive" gets confused with "more correct." SIGKILL (9) gives the process no chance to close open files, release network connections, or save state before dying — the kernel ends it immediately, with no negotiation. SIGTERM (15), kill's default signal, does let the process handle the signal and close in order if it was programmed to. How to spot it: temporary files left half-written, locks that never got released, or a child process left orphaned after the parent got ended with -9 — signs something did not close in order. How to fix it: try kill <PID> first with no number (SIGTERM); give it a few seconds; escalate to kill -9 <PID> only if the process is still alive after that.

Exercises

1. You installed a tool whose executable ended up at /opt/tools/lint-check, with execute permission already confirmed (ls -l shows -rwxr-xr-x). Typing lint-check in the terminal gets you command not found. Your current $PATH is:

/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

What is the root cause, and what two commands do you need — one for this session, another so it survives closing the terminal — to fix it?

See solution

The root cause is that /opt/tools shows up nowhere in $PATH, so the shell never considers that directory when resolving lint-check, even though the executable exists there with correct permissions. For this session: export PATH="/opt/tools:$PATH". So it survives closing the terminal: add that same line to the corresponding rc file and reload it, for example echo 'export PATH="/opt/tools:$PATH"' >> ~/.zshrc && source ~/.zshrc. This works because the shell searches for a slash-free command only in the folders listed in $PATH, in order, and an unpersisted export disappears the moment you close the session that defined it.

2. A teammate, who belongs to the same staff group as you, cannot enter a shared folder you can open with no problem. ls -ld on that folder shows:

drwxr----- 2 ana staff 96 Jul 21 10:15 team-notes

What exactly is your teammate missing, and what command fixes it without granting more access than necessary?

See solution

The group block is r--: the staff group can list the names inside team-notes (r), but cannot traverse the directory (x missing) or write to it (w missing). Since your teammate enters through the "group" category (they are not the owner), they specifically lack the traverse bit. The minimal command is chmod g+x team-notes — it adds only the group's enter permission, with no touch on the "others" category and no write permission nobody asked for. This works because x on a directory is a permission independent from r: you can see the names of what is inside without being able to enter, and vice versa, so you need to grant exactly the missing bit, for the exact class your teammate belongs to.

3. You run lsof -i :5432 and get:

COMMAND    PID  USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
postgres  8842  ana    7u   IPv4 0x...      0t0   TCP *:5432 (LISTEN)

You need to free the port to spin up your own test database. What command do you write first, and why do you not start directly with the most aggressive signal available?

See solution

The first command is kill 8842, with no signal number — that sends SIGTERM (15) by default, the signal that asks postgres to close in order. A database like PostgreSQL, in particular, keeps state files and half-committed transactions; abruptly shutting it down with kill -9 (SIGKILL) takes away its chance to save that state correctly and can leave the database in an inconsistent state that takes longer to repair than the seconds you would save. kill -9 8842 stays as the last resort, only if after a few seconds lsof -i :5432 still shows the process alive. This works because SIGTERM is a signal a well-written process can catch and handle before dying, while SIGKILL gives the process no chance to react.

4. A coworker writes in the team chat: "I had a permissions problem with a project folder, I ran chmod 777 on it and it works now, it's resolved." What would you reply, and what would you do in their place?

See solution

I would reply that "it works" and "it's correctly resolved" are not the same thing: chmod 777 grants read, write, and execute permission to any user on the system, not just whoever actually needed access — on a single-user laptop the immediate risk is low, but on any shared machine (a server, a container with several services) any other account can now read, modify, or replace those files. In their place, I would run ls -ld on the folder to see exactly which bit was missing and for which class (owner, group, or others), and apply the minimum permission needed with symbolic chmod (u+x, g+w, whichever applies) instead of the maximum available. This works because the goal is never "make it stop failing" but "give it exactly the access it needs" — the principle of least privilege you saw at the start of the module applies exactly the same here, with chmod, as it applied with users and with sudo.

Summary and next step

Today you triggered and repaired three real failures using a runbook instead of superstition: a command not found caused by a badly built PATH, a Permission denied that was actually two different causes — a directory with no traverse bit and a file with no execute bit — disguised behind the same message, and a port held by a background process you located with lsof and ended with the right signal. For each one you filled out the same template — symptom, hypothesis, diagnosis, root cause, fix, verification, persistence — because that discipline, not the specific command, is what carries over from one incident to the next.

Before moving on you should be able to: diagnose a command not found telling apart whether the problem is that the file does not exist, that it is missing execute permission, or that its folder is not in $PATH; explain why a directory's x bit and a file inside it's x bit are two independent permissions, even though they produce the same error message; find the process owning any busy port with lsof -i and choose SIGTERM before SIGKILL; and leave any environment fix persistent in your rc file instead of repeating it by hand every session.

What you practiced today — reading before acting, diagnosing before "fixing," and leaving evidence — is exactly the discipline you need for the next module: setting up SSH access with no dependency on passwords requires understanding folder permissions like ~/.ssh with the same precision you understood ~/deploy_tools today, and any real automation you build from here on is going to fail sooner or later for exactly one of these three causes — a badly built PATH, a missing execute bit, or a process left alive holding a resource. You already know how to diagnose all three.

Resources

Command Search and Execution — Bash Reference Manual — how bash resolves a command name against $PATH, the internal hash table, and exit code 127 when the search fails.
chmod(1) — Linux manual page — full reference for symbolic and octal modes, including the distinction between x on a file and x on a directory.
signal(7) — Linux manual page — full table of POSIX signals, with the exact difference between SIGTERM and SIGKILL and which ones a process can catch.
kill(1) — Linux manual page — syntax for the kill command, signals by name and by number, and options like kill -l.
lsof(8) — Linux manual page — how to interpret every column of lsof's output and filter connections with -i.

← Previous7. Configuring your shell: rc files, aliases, and dotfiles Next →1. Introduction: leaving your machine and automating the work