6. Running pytest and the exit code that paints the job

Description

Everything before was preparation: the runner has your code, the correct Python, and the dependencies installed. This lesson is what everything prepared for: running the suite. It's the heart of the module, and it contains the most important idea you take from it: CI discovers and runs your tests exactly the same as you in your terminal, and a single number —the exit code pytest returns on finishing— is what GitHub reads to paint the job green or red. By the end you'll understand the run: pytest step, you'll see the real green output of Reservo's suite (run for real, 11 passed), you'll see a real red run with its exit code, and you'll know what each exit code means.

Here the module's rule shines with all its force. The pytest output you'll see is real —we ran Reservo's suite with Python 3.14.0 and pytest 9.1.1—, and it's identical to what the run: pytest step would produce on the runner, because it's the same command on the same suite. The CI log that illustrates where that output lives is "this is how it would look"; but what that log would contain is exactly what we ran. This lesson is the best example of why the module's honesty works: we don't need a real CI to show you what CI would see, because CI sees the same as your terminal.

Connection to the module: this step closes the chain that lessons 4 and 5 built —it runs on checkout's code, setup-python's Python, and pip install's dependencies—. The exit code you understand here is what lesson 7 reads in the log to put the ✓ or the ✗ of the step and the job. Important boundary: here we provoke a red to see how a failure and its exit code look, but reproducing a failure that happens in CI and not in your local —the environment gap— is module 3. Here we cause the red on purpose and it's the same on both sides; module 3 studies the reds that happen only in CI.

The referee who only gives a thumbs up or down

Imagine a referee in a sport who, at the end of each play, explains nothing: they just give a thumbs up (valid) or a thumbs down (foul). All the analysis —what happened, who touched whom, in which minute— happens in their head and in the replay; but what the scoreboard and the players receive is a single bit: thumbs up or thumbs down. That binary gesture is enough for the game to continue: the scoreboard doesn't need the full analysis to add or not add the point; it needs only the verdict.

pytest is the analyst and the referee at once. In its output —the text it prints— is all the analysis: which tests it ran, which passed, which failed and with what value. But on finishing, besides printing all that, it makes the referee's gesture: it returns to the operating system an exit code, a single number that sums up the verdict. 0 is thumbs up (everything passed); any other number is thumbs down (something went wrong). And GitHub Actions, like the scoreboard, doesn't read the full analysis to decide the job's color: it reads the gesture. If pytest returns 0, the step is green; if it returns anything other than 0, the step is red, and the whole job is painted red.

That's the exact mechanics of how "CI knows whether your tests passed": it doesn't interpret the text, it doesn't count the passed. It runs pytest, and looks at the number pytest leaves on exit. All the rest —the pretty log, the checks— is presentation of that single bit. Understanding this is understanding how CI works inside.

The step, in one line

- run: pytest

That's all. A run type step (a terminal command, lesson 2) that runs pytest, word for word the same command you run on your machine. There's no "CI" version of pytest or magical configuration: it's the pytest you installed in lesson 5, running on the code you brought in lesson 4, with lesson 4's Python. On the runner, this command runs standing at your project's root (where checkout left the code), so pytest does its usual discovery: it looks for test_*.py files, finds the test_ functions, and runs them. Exactly like locally.

In a real project, many write bare pytest in the run: because, unlike an interactive terminal, the runner just installed pytest on the single active Python and there's no ambiguity about which runs. When you run the suite yourself, in your terminal, the no-surprises form is still python3 -m pytest (for the fundamentals guide's reason: it guarantees the pytest of this Python). Both run the same suite; in CI the controlled environment makes bare pytest safe.

Worked example: the green run, for real

Let's run Reservo's suite just as the run: pytest step would run it. This was executed for real in an environment with Python 3.14.0 and pytest 9.1.1, on Reservo's three test folders (prices, refunds, availability).

python3 -m pytest

What to expect:

============================= test session starts ==============================
platform linux -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0
rootdir: /home/runner/work/reservo/reservo
collected 11 items

test_availability.py ....                                                [ 36%]
test_pricing.py ....                                                     [ 72%]
test_refunds.py ...                                                      [100%]

============================== 11 passed in 0.03s ==============================

Read it with what you already know about reading pytest runs, and notice what's identical to your local and what's specific to the runner:

• platform linux — on the ubuntu-latest runner the platform is linux. On your Mac it would say darwin, on Windows win32. It's one of the very few things that change between your machine and CI, and it's one of the reasons a test can pass on one side and fail on the other (module 3's topic). The rest of the header —Python 3.14.0, pytest-9.1.1— is identical because setup-python and pip install pinned it that way on purpose.
• rootdir: /home/runner/work/reservo/reservo — the runner's working folder, where checkout left the code. On your machine it would be your local path; the number that matters is below.
• collected 11 items — discovery in action, the same as locally: pytest found Reservo's 11 tests (4 availability, 4 pricing, 3 refunds) without you telling it which. CI keeps no test list; it discovers them on every run, like you.
• The progress line —test_availability.py ...., test_pricing.py ...., test_refunds.py ...— one dot per test that passes, file by file. Eleven dots, eleven green tests.
• 11 passed in 0.03s — the verdict. Eleven tests, all passed, in three hundredths of a second (Reservo is pure logic, it flies).

Now the piece that connects with CI, the one not visible in the text but the one GitHub reads. When pytest finishes this green run, it returns exit code 0. We can verify it in the terminal by asking for the exit code of the last command:

python3 -m pytest -q > /dev/null; echo "exit code: $?"

exit code: 0

0 is thumbs up. On the runner, GitHub sees that 0 and paints the run: pytest step green, and since it's the last step of the job, the whole job goes green. That's the entire connection between "the tests passed" and "the pipeline is green": pytest returned 0, GitHub read it. Without interpreting the text, without counting the passed.

Worked example: the red run and its exit code

To see the other side of the thumb, let's provoke a failure —in the same honest way as the fundamentals guide: we break the code on purpose—. In reservo/pricing.py, we change the pro discount from 20% to 25% (a plausible bug: someone "improves" the promotion without warning). This was also executed for real:

============================= test session starts ==============================
platform linux -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0
rootdir: /home/runner/work/reservo/reservo
collected 11 items

test_availability.py ....                                                [ 36%]
test_pricing.py .F..                                                     [ 72%]
test_refunds.py ...                                                      [100%]

=================================== FAILURES ===================================
_____________ test_pro_member_gets_twenty_percent_off_the_subtotal _____________

    def test_pro_member_gets_twenty_percent_off_the_subtotal():
>       assert price_cents(FOCUS, BETO, 3) == 6000
E       AssertionError: assert 5625 == 6000
E        +  where 5625 = price_cents(Room(id='r1', name='Focus', capacity=1, hourly_cents=2500), Member(id='m2', name='Beto', tier='pro'), 3)

test_pricing.py:17: AssertionError
=========================== short test summary info ============================
FAILED test_pricing.py::test_pro_member_gets_twenty_percent_off_the_subtotal - AssertionError: assert 5625 == 6000
========================= 1 failed, 10 passed in 0.03s =========================

And its exit code:

python3 -m pytest > /dev/null 2>&1; echo "exit code: $?"

exit code: 1

Read the run and the number together, because it's where everything sets:

• The progress line test_pricing.py .F.. already draws the damage: an F among the dots, the pro discount test that failed. The other ten tests stay green (the three files, minus that one).
• The diagnosis is real and precise: assert 5625 == 6000. With a 25% discount, 7500 − 1875 = 5625, not the expected 6000. pytest shows you the exact value that went wrong, the same in CI as locally —assert rewriting doesn't change because you're in the cloud—.
• The verdict 1 failed, 10 passed and, the key piece, the exit code 1. Any failure makes pytest return 1 (thumbs down). On the runner, GitHub sees that 1, paints the run: pytest step red with a ✗, and the whole job is painted red. The push (or pull request) gets marked red, and there's the alarm module 1 promised: the pipeline warns you, without anyone having to remember to run the tests.

Once seen, we restore the code (back to 20%) and the suite returns to 11 passed and exit code 0. That cycle —green gives 0, red gives something other than 0— is the whole CI machine. The job's color is a function of a number.

The pytest exit codes you'll encounter

pytest doesn't return only 0 or 1; it has a small catalog of codes, and it's worth knowing the three that appear most, because each one tells CI a different story:

Exit code	What it means	How the job looks in CI
0	All tests passed.	Green ✓
1	At least one test failed.	Red ✗
5	No tests were collected (no tests ran).	Red ✗

The 0 and the 1 you already saw. The 5 is treacherous and deserves attention: it means pytest ran but found no test to run. We verified it for real by running pytest in an empty folder:

============================= test session starts ==============================
platform linux -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0
rootdir: /home/runner/work/reservo/reservo
collected 0 items

============================ no tests ran in 0.00s =============================

exit code: 5

Why the 5 matters so much in CI: if pytest returned 0 when it finds no tests, a workflow with the tests misplaced or misnamed would pass green without having tested anything —the worst possible lie, a pipeline that says "all good" without having checked anything—. pytest avoids that trap by returning 5, different from 0, so the job is painted red and forces you to look. If you ever see a red CI with "no tests ran" and exit code 5, it's not that your tests failed: it's that CI didn't find your tests (wrong path, misplaced names, an absent checkout). The 5 is pytest protecting you from a false green.

Common mistakes

Believing CI "reads" how many tests passed in the text (wrong mental model). What happens: someone imagines GitHub parses the 11 passed line to decide the color, and gets confused when something doesn't add up. Why it happens: the text is what's visible, so you assume it's what's read. How to spot it: if you try to explain why a job is red by talking about the text instead of the exit code, you have the model backwards. How to fix it: remember that GitHub reads the exit code, not the text. The text is for the human; the number is for the machine. A step is green if its command returned 0, and red if it returned anything else —regardless of what the text says—.

Confusing an exit code 5 (found no tests) with a test failure (wrong diagnosis). What happens: CI is red, someone sees "no tests ran", and starts looking for which test failed —when none failed, the problem is it found none—. Why it happens: red is automatically associated with "a test failed", but the 5 is a red of another nature. How to spot it: collected 0 items and no tests ran with exit code 5 mean "I found no tests", not "a test broke". How to fix it: when you see the 5, check why CI didn't find your tests —did checkout bring the code?, do the names follow the test_* convention?, does pytest run from the correct folder?—. It's a discovery problem, not a logic one.

Suppressing the exit code and forcing the job green (silencing the alarm). What happens: someone, fed up with seeing red, writes run: pytest || true —the || true forces the command to "finish fine" no matter what—, and the job stays green even if the tests fail. Why it happens: "I want CI to stop bothering me" gets confused with "I want to fix the problem". How to spot it: if your pytest step has a || true, a continue-on-error, or anything that ignores the exit code, your CI no longer protects anything —it'll always be green—. How to fix it: never silence pytest's exit code; that number is the protection. A CI that's always green is exactly as useful as having no CI. If the red bothers you, fix the cause (the test or the code), don't turn off the alarm.

Exercises

Exercise 1 — Predict the job's color. For each of these three pytest runs on the runner, say which exit code it returns and what color the job is in CI (green or red), and why: (a) 11 passed in 0.03s; (b) 2 failed, 9 passed in 0.04s; (c) no tests ran in 0.00s.

See solution

• (a) 11 passed → exit code 0 → green job. All tests passed; pytest returns 0 (thumbs up) and GitHub paints the step green.
• (b) 2 failed, 9 passed → exit code 1 → red job. It's enough for one test to fail for pytest to return 1; two failing and nine passing doesn't change the binary verdict: there were failures, the number is 1, the job is red.
• (c) no tests ran → exit code 5 → red job. pytest found no test; it returns 5 (different from 0) precisely so the job is red and not a false green. It's a red of "I found no tests", different in cause from (b), but red all the same.

The thread: the color depends only on whether the exit code is 0 or not. 0 is the only green; 1 and 5 (and any other non-zero) are red. The text explains why, but the number decides the color.

Exercise 2 — Diagnose the false danger. A teammate opens their CI and sees it red. The pytest step's log shows this. They're convinced they broke some test and have spent half an hour reviewing their pricing logic. What would you tell them?

collected 0 items

============================ no tests ran in 0.00s =============================

Exit code: 5

See solution

I'd tell them to stop looking for a broken test, because none broke: the exit code 5 and the collected 0 items mean pytest found no test to run. It's not a logic failure; it's a discovery failure. Their pricing logic may be perfect —pytest never even got to run it, because it found no tests—.

The typical causes of the 5, in order of probability to check:

• The checkout didn't bring the code (or the step is missing), so the test folder isn't on the runner. rootdir and the absence of files would give it away.
• The tests are misplaced or misnamed: files that don't start with test_, functions that don't start with test_, or a folder pytest isn't looking at.
• pytest runs from the wrong folder, without the tests beneath it.

The fix depends on which one it is, but the correct diagnosis saves the half hour: red with exit code 5 isn't "a test failed", it's "I found no tests". Reviewing the logic of a test that never even ran is looking in the wrong place. (That CI is red here instead of a misleading green is, in fact, pytest protecting them: a pipeline that passed green without having tested anything would be much worse.)

Exercise 3 — Explain why you don't silence it. A teammate, tired of CI turning red while they work, proposes changing the step to run: pytest || true "so the pipeline doesn't block while I fix things". Explain to them with the exit-code mechanics why that empties CI of meaning, and what they should do instead.

See solution

The || true makes the step always finish with exit code 0, no matter what happens with the tests: pytest can return 1 (failures) or 5 (no tests), but the || true replaces it with a 0 before GitHub reads it. And since GitHub decides the job's color only by the exit code, the step —and the job— will stay green forever, even if half the tests are broken.

That empties CI of meaning because its only job is to warn you when something breaks, and it warns by turning red. A CI that can't turn red warns of nothing: it's exactly as useful as having no CI, with the aggravating factor that it gives a false sense of security —the green lies—. The disconnected smoke alarm looks the same as the one that works, until the fire.

What they should do instead: let CI turn red and fix the cause. If a test fails, fix the test or the code; if the red is because they're mid-task, that's exactly the signal that the change isn't ready to share yet —the point of CI is that broken code isn't shared—. If they want to iterate fast without waiting for CI, run the tests locally (python3 -m pytest -k "what-im-fixing") while they work, and push when they're green. The solution is never to silence the number that is the protection; it's to attend to what the number tells you.

Summary and next step

In this lesson you reached the heart of the module: the run: pytest step and the exit code. You understood the central idea —CI discovers and runs your suite exactly the same as your terminal: same command, same suite, same discovery— and you saw the real green output of Reservo (11 passed, run for real with Python 3.14.0 and pytest 9.1.1). Above all, you understood the mechanics by which "CI knows how it went for you": pytest, like a referee, returns an exit code on finishing, and GitHub reads it to paint the job. 0 is all green; 1 is at least one test failed (you saw it real, with the 25% bug and its assert 5625 == 6000); 5 is found no tests —a red that protects you from a false green—. The job's color is a function of a single number, not of the text.

Before moving on you should be able to: write the step that runs pytest; explain how the exit code determines the job's color; name what 0, 1, and 5 mean; distinguish a red of "test failed" (1) from one of "found no tests" (5); and argue why silencing the exit code destroys CI's value.

You now have the complete workflow end to end —when, where, and all the steps of the what—. What's missing is learning to read what it produces when it runs. In lesson 7 we open the CI log: how it's organized (jobs that contain collapsible steps, a ✓ or ✗ for each), where exactly pytest's output lives inside the step that generated it, how a step's ✗ corresponds to the non-zero exit code you just understood, and how the status badge is hung on the README —that green check that tells the world, at a glance, that the suite passes—. It's the lesson that teaches you to interpret the result of the pipeline you built.

Resources

• How to invoke pytest (pytest documentation) — the official reference for how pytest is run and, at the end, the "Possible exit codes" section with the complete list of exit codes (0 to 5). The exact source of this lesson's table.
• pytest Get Started — the basic flow of running pytest and reading its output, the same one CI produces. Useful for reinforcing reading a green run and a red one.
• Environment variables and job status (GitHub Actions documentation) — how GitHub Actions determines the success or failure of a step and a job from its commands' exit code. The "GitHub" side of the mechanics this lesson explained from the "pytest" side.