Module 3: Reproducing A Ci Failure Locally

2. The symptom: red here, green there (and its causes)

Next →

Description

In the previous lesson you named the ghost and the rule that hunts it —reproduce before fixing—. This lesson opens the symptom wide: when CI says red and your machine says green about the same commit, what can be different between the two, concretely? You're going to come out with a catalog of suspects: the list of the things that differ between a CI runner and your computer and that produce this disagreement. Having that list in your head turns panic ("I don't understand, it works on my machine!") into an ordered investigation ("let's see which of these six things is different this time").

You're also going to learn the correct mindset toward the symptom, which is more important than the list: when the same code gives two results, the test almost never lies; the environment differs. That phrase saves you the most expensive mistake —disabling a test that's actually detecting a real difference— and points you from the first second toward where the problem really is.

Connection to the module. Lesson 1 gave the map and the discipline. This one gives the dictionary of causes: the six or seven types of environment gap that produce "red here, green there". Lesson 3 goes deeper into what the environment is and why the two machines are different worlds; lessons 4, 5, and 6 attack the most common causes one by one (dependencies, dirty environment, variables and hidden differences). Here we build the complete list so the following lessons have somewhere to hang each solution. We continue with the Reservo suite and its pytz failure.

The analogy: the detective and the list of suspects

A detective who arrives at a scene doesn't interrogate at random. They have a mental list of suspect categories —who had motive, who had opportunity, who was nearby— and they rule out. Without that list, each case would be starting from scratch, staring at nothing. With it, the investigation is a procedure: check each category, see which fits, follow that lead.

A "CI red, local green" failure has a surprisingly short and stable list of suspects. It's almost always one of these: a different Python version, a dependency with another version, an environment variable, the order the tests ran in, the machine's time zone, or a file that only exists on your disk. Six usual suspects. When you run into the symptom, don't stare at nothing: go through the list. "Same Python version? Same library versions? Any variable I have and the runner doesn't?". That methodical run-through is what separates a lost afternoon from a twenty-minute investigation.

Said in one phrase, which is the one that governs the lesson:

The disagreement between CI and your machine about the same code is always explained by an environment difference. The catalog of those differences is short and known; reproducing the failure is going through it until you find which one applies.

The environment-gap catalog

Here's the list of suspects, each with its mechanism and a sign to recognize it. Don't memorize them like someone memorizing a table; understand why each one moves the result, and the list will stick on its own.

1. A different Python version

The CI runner runs, say, Python 3.13, and you have 3.14 (or the reverse). Between Python versions things change: a standard-library function behaves differently, a new syntax doesn't exist in the old one, the order of a dictionary or a set can differ in edge cases, an error message changes text. A test that depends on any of those details passes on one version and fails on another. Sign: the CI log, in its first pytest line, says Python 3.13.x, and you run python --version and see 3.14.0. That line —which almost no one reads— is the first place to look.

2. A dependency with another version

This is the queen of causes, and the one Reservo suffers in this module. Your machine has a version of a library installed —say pytz 2022.1, which you installed a year ago and never updated—; CI does a fresh install on every run and grabs the latest —pytz 2026.3.post1—. If the library's behavior changed between the two versions, the same test gives different results. Sign: the failure involves a call to a third-party library (formatting, dates, time zones, serialization, network), and pip show <lib> on your machine shows a version different from the CI log's. We dedicate lesson 4 entirely to it.

3. An environment variable

Your shell has a variable exported —RESERVO_TAX_PERCENT=16, DATABASE_URL=..., TZ=America/Mexico_City— that your code reads with os.environ. The CI runner starts clean and doesn't have it (or has it with another value). The code takes a different path depending on the value, and the test changes color. Sign: the failure disappears if you remove a variable from your shell, or if your test reads something from configuration. Variables are especially treacherous because they're invisible in the code and in requirements.txt; they live in your shell and in the runner's config. We hunt them in lesson 6.

4. The order of the tests

pytest runs the tests in some order; some plugins (like pytest-randomly) shuffle them. If two tests share state by accident —one leaves a global variable modified, a file written, an entry in a module dict— the one that runs second can pass or fail depending on who ran before. CI and your machine can collect the tests in a different order (because of the file system, because of a randomization seed) and so one sees the failure and the other doesn't. Sign: the failure changes if you run the tests in another order, or if you run the culprit test alone and then it passes. The in-depth diagnosis of this —"order coupling"— lives in the sibling guide test-failure-diagnosis-guide; here it's enough to recognize it as a suspect and to know that replicating the order seed is part of reproducing.

5. The time zone (and the time-zone data)

The machine has a system time zone, and date libraries carry a time-zone database that gets updated over time. A test that builds or interprets a "local" date depends on both. If your machine is on America/Mexico_City and the runner on UTC, or if your pytz carries the old data (with daylight saving) and the runner the new one (without it), the same instant falls at different hours. Sign: the failure involves dates, times, datetime, offsets, or zone names, and the expected number is "shifted" one or two hours. It's exactly Reservo's case: old pytz says 16:00, new pytz says 15:00. Note this suspect overlaps with #2 (a dependency with another version), because the tz data lives inside a dependency —that's what makes Reservo's case so illustrative: it's both suspect #2 and #5—.

6. A file that only exists on your disk

Your code or your test reads a file —a .env, a test-data CSV, a config file, a fixture— that's on your machine but wasn't pushed to the repository (because of a .gitignore, or because you created it by hand and forgot to commit it). On your machine the file exists and the test passes; on the runner, which only has what's in git, the file doesn't exist and the test fails (often with a FileNotFoundError). Sign: the CI failure is a FileNotFoundError or a No such file or directory, and the file it mentions exists on your disk but git status doesn't list it or git ls-files doesn't include it. Related: hardcoded absolute paths (/Users/yourname/project/data.csv) that only exist on your machine.

A seventh, cross-cutting one: the working directory and the accumulated state

There's a more diffuse suspect worth keeping in mind: where you ran the tests from and what accumulated on your machine. If you run pytest from the project root, imports and relative paths resolve differently than if you run from a subfolder. And your machine has been accumulating things for months —packages installed "temporarily" and never uninstalled, caches, an old conftest.py, a forgotten .pth— that the clean runner doesn't have. This seventh suspect is the topic of lesson 5 (the clean venv), because the best way to rule it out is to start from scratch like CI does.

Why the test almost never lies

Of the whole catalog, the most valuable lesson isn't the list but the attitude. When you see red in CI and green locally, your brain jumps to the explanation that suits you most emotionally: "the test is wrong", "CI is slacking", "it's a false positive". It's tempting because disabling a test is faster than investigating, and because it hurts less to blame the tool than your own code. But think about it coldly: the test is a piece of deterministic code. Given the same inputs —same code under test, same versions, same variables— it always produces the same result. If it produces two different results, then the inputs weren't the same. And "the inputs" is, precisely, the environment.

That's why the rule: when the same commit gives red on one side and green on the other, the suspect is the environment, not the test. The test is doing exactly its job —detecting that, under CI's conditions, the code produces a result you didn't expect—. Disabling it is killing the messenger. In Reservo's case, the test that fails in CI is right: with the current time-zone data, the booking really starts at 15:00 local, not at 16:00. The "16" was an old assumption. The test isn't broken; it's warning that the world changed. If you'd marked it with skip, you'd have buried that signal.

There's an honest nuance worth saying: there's an exception to "the test almost never lies", and it's the flaky test —one that fails intermittently even in the same environment, because it depends on the clock, on chance, on the network, or on order—. That's a different problem (module 7 of this guide deals with it, and its diagnosis the sibling guide). But even there, the cure isn't to disable it blindly: it's to understand why it's intermittent. The general rule holds: before blaming the test, suspect the environment and reproduce.

Worked example: reading the red like a detective

Let's go back to Reservo's failure and practice going through the catalog on a real log. This is the failure summary as CI would paint it (the format is pytest's; on a GitHub Actions runner it appears inside the log of the step that runs the tests):

============================= test session starts ==============================
platform linux -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0
collected 1 item

test_localtime.py::test_summer_booking_starts_at_16_local FAILED          [100%]

=================================== FAILURES ===================================
____________________ test_summer_booking_starts_at_16_local ____________________

>       assert local_start_hour(a_booking(), "America/Mexico_City") == 16
E       AssertionError: assert 15 == 16

test_localtime.py:21: AssertionError
=========================== short test summary info ============================
FAILED test_localtime.py::test_summer_booking_starts_at_16_local - assert 15 == 16

Now go through the catalog with the information you have. Python version? The log says Python 3.14.0; if you also run 3.14.0, ruled out (suspect #1 out). A missing file? There's no FileNotFoundError; it's an AssertionError of a numeric value. Ruled out (#6 out). Test order? Only 1 test was collected, so there's no interaction between tests. Ruled out (#4 out). Dates and times? Yes: the test is called ..._starts_at_16_local, passes a time zone ("America/Mexico_City"), and the obtained value (15) is shifted exactly one hour from the expected (16). That "shifted one hour" is the classic fingerprint of suspect #5 (time zone) and, since the tz data lives in pytz, also of #2 (dependency version). Environment variable? Possible, but the pattern of "one hour of difference in a time-zone calculation" points much more strongly to the tz/dependency.

In thirty seconds, without touching the code, the catalog took you from "I don't understand anything" to "this smells like a pytz version". The next step —confirming it by comparing the version you have with the one CI installed, and reproducing— is what lessons 4 and 5 build. But notice what you just did: you turned a baffling red into a concrete, verifiable hypothesis, just by having the list of suspects in your head and reading the log methodically.

What to expect. When you really have the two environments in front of you, the confirmation looks this sharp. On the machine with the old pytz:

$ python -c "import pytz; print(pytz.__version__)"
2022.1
$ python -m pytest test_localtime.py -q
1 passed in 0.02s

And in an environment with a fresh install (the latest pytz), identical in everything else:

$ python -c "import pytz; print(pytz.__version__)"
2026.3.post1
$ python -m pytest test_localtime.py -q
1 failed in 0.04s

Two versions of a single dependency, two colors. The catalog pointed to the correct suspect, and the version comparison confirmed it. That's reproducing on the right track: not guessing, but going through the list and verifying.

Common mistakes

Treating the symptom as a unique, one-off mystery. What happens: every time a "red here, green there" appears you live it as an unprecedented case and start investigating from scratch, looking at the code by eye. Why it happens: the symptom is scary and panic erases the method. How to spot it: if your first move is to reread the function's code instead of comparing the two environments, you're without a list. How to fix it: memorize the catalog of six suspects and go through it always in the same order. The vast majority of these failures fall into one of those categories; the method takes you to the suspect in minutes.

Reading only the AssertionError and skipping the log header. What happens: you go straight to the failure line and never look at pytest's first line (platform ... Python 3.x.y, pytest-...). Why it happens: the header seems like startup noise. How to spot it: if you don't know which Python version CI used, you didn't read the header. How to fix it: the header is CI's environment ID card —Python version, pytest version, platform—. It's the first place to compare against your machine. Always read it.

Assuming "green on my machine" means "the code is fine". What happens: since it passes on your machine, you conclude the code has no problem and the CI red is CI's doing. Why it happens: we trust more what we see with our own eyes (your green terminal) than a remote log. How to spot it: if your conclusion is "the code is fine, the problem is CI", without having compared environments, you made this leap. How to fix it: "green on your machine" only means "the code passes under your machine's conditions". CI runs under other conditions, and the red says that under those the code fails. Both are real data; the question isn't who's right, but what differs between the two.

Exercises

Exercise 1 — Classify the suspect. For each CI failure (green on your machine), say which of the catalog's suspects is the most likely and why. (a) In CI: ModuleNotFoundError: No module named 'reservo.localtime', and the file exists on your disk. (b) In CI: AssertionError: assert 15 == 16 in a local-time calculation. (c) In CI: SyntaxError on a line that uses match/case. (d) In CI: a test passes when run alone, but fails when run after another.

See solution
  • (a) Suspect #6 (a file that only exists on your disk). The module exists on your machine but the runner doesn't have it: probably reservo/localtime.py wasn't committed (check it with git ls-files). It could also be a packaging/imports problem (suspect #7, working directory), but the first suspicion with a module "that exists locally" is that it's missing from git.
  • (b) Suspects #5 and #2 (time zone + dependency version). The value shifted exactly one hour in a time-zone calculation is the classic fingerprint; the tz data lives in a dependency, so it's both tz and library version. It's Reservo's case.
  • (c) Suspect #1 (Python version). match/case exists since Python 3.10; a SyntaxError there means the runner runs a version older than yours.
  • (d) Suspect #4 (test order). Passing alone but failing in sequence is the definition of order coupling: some shared state between tests. (The in-depth cure is from the diagnosis guide; here it's enough to recognize it.)

Exercise 2 — The hidden suspect. A test reads the environment variable RESERVO_TAX_PERCENT to compute a total with tax. It passes on your machine and fails in CI. Explain the exact mechanism —why your machine and the runner differ— and why this suspect is harder to see than a dependency version.

See solution

The mechanism: in your shell you have RESERVO_TAX_PERCENT exported (for example, export RESERVO_TAX_PERCENT=16 in your .zshrc or from a previous session), so when the code does os.environ.get("RESERVO_TAX_PERCENT", "0") it gets "16" and the total includes the tax the test expects. The CI runner starts with a clean environment and doesn't have that variable, so the code gets the default "0", the total carries no tax, and the test's expected number doesn't add up.

Why it's harder to see than a dependency version: a version lives in requirements.txt and in pip freeze —it's visible and comparable between the two machines—. An environment variable lives in your shell and in the runner's configuration, not in any file of the repository; it's invisible when reading the code or the dependencies. You can look at the whole project and not find it, because the value that breaks the symmetry isn't in the project, it's in the air of each machine. That's why variables are hunted by comparing the environments with env/printenv, not by reading the repo (lesson 6).

Exercise 3 — Defend the test. A teammate proposes: "This local-time test is a pain; it fails in CI and passes locally. Let's put @pytest.mark.skip on it and move on." Give two concrete reasons why skipping the test is the worst option here, and what you'd propose instead.

See solution

Two reasons not to skip it:

  1. The test is right, it's not broken. With the current time-zone data, the 21:00 UTC booking really starts at 15:00 in Mexico City (which abolished daylight saving time). The 16 was an assumption that aged. Skipping the test buries a correct signal: that the code's expected number went stale. If you skip it, at some point a user will see the wrong hour and no one will find out.
  2. You turn off a protection forever for a one-day problem. A skip is rarely removed; the test stays dead and the rule it covered —"the local time is computed correctly"— stops being protected. Any future bug in that calculation will pass undetected.

What to propose instead: reproduce the failure (compare CI's pytz version against the local one, set up a clean venv with CI's version, run the same command), confirm it's a time-zone-data change, and then decide with knowledge —most likely, correct the expected number to 15 and pin pytz so CI and local coincide reproducibly—. That fixes the cause; the skip only hides the symptom.

Summary and next step

In this lesson you built the environment-gap catalog: the short and stable list of things that differ between CI and your machine and produce "red here, green there". A different Python version; a dependency with another version (the queen of causes); an invisible environment variable; the order of the tests; the time zone and its data; a file that only exists on your disk; and, cross-cutting, the working directory and your machine's accumulated sediment. You practiced going through that list on Reservo's real log and arriving, in thirty seconds and without touching code, at a concrete hypothesis: "this smells like a pytz version".

And you recorded the attitude that avoids the most expensive mistake: when the same code gives two results, the test almost never lies; the environment differs. Disabling the test is killing the messenger; Reservo's test that fails in CI is right, and skipping it would have buried a correct signal.

Before moving on you should be able to: recite the six suspects of the catalog with their mechanism; read the header of a pytest log to extract the Python and pytest version; recognize the fingerprint of a time-zone difference (a value shifted one hour); and argue why you don't disable a test that disagrees between CI and local.

What's next is understanding what that "environment" we talk about so much is, piece by piece, and why the CI runner and your machine are, by nature, two different worlds. Lesson 3 dissects the environment gap —Python, dependencies, variables, system, tz, files, directory— and shows you with real output a hidden difference in action, so the idea stops being a list and becomes something you can see and touch.

Resources

← Previous1. Module introduction: CI red, your machine greenNext →3. The environment gap, up close