Module 3: Reproducing A Ci Failure Locally

1. Module introduction: CI red, your machine green

Next →

Description

In module 2 you set up your first pipeline: a GitHub Actions workflow that, on every push, takes a copy of your code, installs Python, installs your dependencies, and runs pytest. Since then every change you push gets an automatic verdict —green bar or red bar— without you having to remember to run the tests. It's a guardian that doesn't sleep.

This module is about the day that guardian and you don't agree. You push a change, CI turns red, you open the same project on your machine, run the same suite, and it comes out green. The failure exists —there's the red log, with its AssertionError—, but you can't touch it, because on your computer it doesn't happen. It's the ghost bug: real on one side, invisible on the other. And it is, without exaggeration, one of the situations that steals the most time from a team, because the instinctive reaction —"CI is broken", "it's a false positive", "I'll run it again to see if it passes"— is almost always the wrong one.

By the end of the module you'll have a cold and repeatable method to turn that ghost into a failure that also happens on your machine, whenever you want, as many times as you want. Because that's the iron rule that governs everything that follows: a failure you can't reproduce you can't fix. Reproducing isn't an optional step or a luxury; it's the entryway. As long as the failure only lives in CI, any "fix" you attempt is guessing blindly: you change something, push, wait five minutes for the pipeline to run, and pray. When the failure happens on your machine, the loop closes: you change, run, see the result in seconds, and know with certainty whether you fixed it.

Connection to the module. This lesson opens the module: it gives you the map, the vocabulary, and the discipline before going down to the details. Lessons 2 and 3 diagnose the symptom (red here, green there) and its root cause (the environment gap). Lessons 4, 5, and 6 attack the three concrete sources of that gap —unpinned dependencies, a dirty environment, and hidden variables and differences—. Lesson 7 pulls it all together into a step-by-step method, and lesson 8's mini-project has you reproduce and fix a real Reservo failure from start to finish. All on the Reservo suite you already know, and with a demonstration I ran for real on my machine —not invented— where the same test passes with one version of a library and fails with another.

The analogy: the recipe that only turns out for you

Think of a friend who gives you the recipe for a bread that comes out perfect for her. You follow it to the letter —the same grams, the same steps, the same oven "at 180 degrees for 25 minutes"— and it comes out raw inside for you. You send her a photo, she swears it comes out fine for her, and you swear you followed every line. Who's lying? No one. The recipe —the code— is identical. What differs is the environment: her oven runs hotter than yours, she lives at sea level and you at 2000 meters altitude (where water boils at a lower temperature), her flour has more gluten than the one you buy. The recipe isn't wrong; the world around the recipe is different in each kitchen.

A "CI red, local green" failure is exactly that. The code is the same —byte by byte, it's the same commit—. What differs is the kitchen: the Python version, the versions of the installed libraries, the environment variables, the machine's time zone, the files on the disk. Your instinct as an experienced cook isn't to throw the recipe in the trash ("this test is broken, I'll delete it"); it's to ask yourself what's different between my kitchen and hers. Reproducing the failure is, no more and no less, cooking the recipe in a kitchen just like CI's: same oven, same altitude, same flour. When you get it to come out raw for you too, you stopped arguing about whether the recipe is wrong and started fixing the real problem.

It's worth saying it directly, because it's the thesis of the whole module:

When CI and your machine disagree about the same code, the test almost never lies: the environment differs. Reproducing the failure is closing that environment gap until the red also appears on your machine. Only then —not before— can you fix it.

What "reproducing" is and what it isn't

It's worth separating two things people mix and that in this guide live in different modules and guides: reproducing a failure and diagnosing it.

Reproducing is getting the failure to happen at will on your machine. It doesn't tell you why it fails; it puts the failure in your hand, live, so you can work with it. It's an environment problem: making your kitchen resemble CI's in what matters. Reproducing is answered with questions like "which Python version did CI use?", "which version of that library did it install?", "which variables did it have set?", "which exact command did it run?". This module is only about this.

Diagnosing is understanding why the code produces the wrong result once you already have it failing in front of you: isolating the culprit line, using the debugger, putting a breakpoint(), reducing the test to the minimum that reproduces the error. That's a craft of its own, with its own tools, and it lives in the sibling guide test-failure-diagnosis-guide (pdb, isolate, bisect). We don't go there here. Our boundary is sharp: this module ends the instant the CI failure also happens on your machine. That's the "handoff": with the failure reproduced, the environment gap is closed, and the question changes from "why doesn't it happen to me?" to "why does the code do this?" —and that second question is answered in the other guide—.

Why separate them so sharply? Because mixing them is the number-one source of lost hours. People try to diagnose a failure they can't yet reproduce: they start reading the code by eye, theorizing about the cause, changing lines "just in case" and pushing them to see if CI changes color. It's debugging through a five-minute-per-attempt pipeline, without being able to put a print, without being able to inspect anything. It's slow, frustrating, and almost always useless. The discipline is the reverse: first reproduce, then diagnose. Get the failure on your machine —where you have all your tools— and only then start to understand why. This module gives you the first half, which is the one that unlocks the second.

The case that runs through the module: Reservo and a time zone that changed

So this isn't abstract, the whole module revolves around a concrete, real, and reproducible Reservo failure —the coworking meeting-room booking app you've been using in the testing guides—. Recall its pieces: Room, Member, Booking (with its price_cents field, money always in integer cents), price_cents(room, member, hours), refund_cents(booking, price_paid_cents, now), the Calendar. And its anchor numbers: a basic member pays 3 hours of the Focus room at 7500 cents; a pro pays 6000 (20% discount); a refund 72 hours ahead returns 6000, 36 hours ahead returns 3000, and 12 hours ahead returns 0.

Reservo grew and now has members in several cities. To show each one their bookings in their local time, someone added a little function that, given the UTC time a booking starts, computes the wall-clock time in the member's time zone —for example, for a member in Mexico City—. They leaned on a popular time-zone library, pytz, and wrote a test with an expected number. The test passed on their machine. They pushed it. And weeks later, without anyone touching that code, CI turned red.

That's our ghost. The code didn't change; neither did the test. What changed was a piece of the environment —the version of pytz CI installed in its most recent run—. And behind it there's a real-world fact that makes it all juicier: Mexico abolished daylight saving time in October 2022. Old versions of pytz still believed Mexico City moved the clock forward in summer (UTC−5); the new ones already know it doesn't (UTC−6 all year). A booking at 21:00 UTC on a July day of 2023 falls at 16:00 local time with the old pytz and at 15:00 with the new one. The test expected 16. With the old library —the one the dev had installed— it passed. With the new one —the one CI installed fresh— it fails.

Throughout the module we're going to dissect this case: why the two machines ended up with different versions of pytz (lesson 4), how to set up a clean venv that replicates CI's (lesson 5), what other hidden differences —like an environment variable— produce the same symptom (lesson 6), and the complete method to reproduce the red at will (lesson 7). And it's not a fairy tale: I ran it for real in two environments, with Python 3.14.0 and pytest 9.1.1, and you're going to see the two outputs —the green and the red— with your own eyes in the next lesson.

A first look at the ghost

Before closing the introduction, look at the phenomenon with real output, so it doesn't stay a promise. This is the same Reservo test run in two environments that differ in a single thing: the installed version of pytz. Everything else is identical —the same test file, the same Reservo code, the same machine, the same Python 3.14.0, the same pytest 9.1.1—.

The test, just as the dev wrote it:

# test_localtime.py — the test that passes on one machine and fails on the other
from datetime import datetime

import pytz

from reservo.models import Booking
from reservo.localtime import local_start_hour

UTC = pytz.utc
SUMMER_START = UTC.localize(datetime(2023, 7, 15, 21, 0))  # 21:00 UTC, a summer day


def a_booking():
    return Booking(id="bk-1", room_id="r-focus", member_id="m-1",
                   start=SUMMER_START, end=SUMMER_START,
                   status="confirmed", price_cents=6000)


def test_summer_booking_starts_at_16_local():
    # The dev wrote it expecting summer in CDMX = UTC-5 (daylight saving time).
    assert local_start_hour(a_booking(), "America/Mexico_City") == 16

What to expect. In the dev's environment (with pytz 2022.1, a version they'd had installed for a while), the suite comes out green:

$ python -m pytest test_localtime.py -v
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0
collected 1 item

test_localtime.py::test_summer_booking_starts_at_16_local PASSED         [100%]

======================== 1 passed, 1 warning in 0.02s =========================

And in an environment with a fresh install (which grabbed the latest pytz, 2026.3.post1) —which is exactly what the CI runner does on every run—, the same suite comes out red:

$ python -m pytest test_localtime.py -v
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0
collected 1 item

test_localtime.py::test_summer_booking_starts_at_16_local FAILED         [100%]

=================================== FAILURES ===================================
____________________ test_summer_booking_starts_at_16_local ____________________

    def test_summer_booking_starts_at_16_local():
        # The dev wrote it expecting summer in CDMX = UTC-5 (daylight saving time).
>       assert local_start_hour(a_booking(), "America/Mexico_City") == 16
E       AssertionError: assert 15 == 16

test_localtime.py:21: AssertionError
=========================== short test summary info ============================
FAILED test_localtime.py::test_summer_booking_starts_at_16_local - assert 15 == 16
========================== 1 failed in 0.04s ===========================

Read it slowly. The code is identical. The test is identical. The only difference between the two runs is the version of one library, and that difference moves the result from 16 to 15, and with it the color from green to red. No test is "broken". The new pytz is right: in 2023, Mexico City no longer had daylight saving time, so 21:00 UTC really is 15:00 local. The test aged with an assumption that stopped being true, and only one of the two machines had the library updated enough to notice.

Notice a detail that's also part of the gap, even though today it's not the protagonist: my log says platform darwin because I ran it on macOS. The CI runner would say platform linux. That line, which almost no one looks at, is a reminder that the two kitchens are never identical by default —you have to make them coincide in what matters—. That "making them coincide" is, in one phrase, the whole module.

Deep dive: the economics of the feedback loop

It's worth putting numbers to why reproducing first isn't a whim of method but pure economics. A pipeline's value is its feedback loop: you change something, and a machine tells you if you broke something. That loop has a cost per round —the time between "I push a change" and "I know the result"—, and that cost decides how you work. In CI, one round costs the whole pipeline: commit, push, wait for the runner to start, install dependencies, and run the suite. Easily three to five minutes, sometimes ten or fifteen in large projects. On your machine, one round costs what pytest takes: for the Reservo suite, fractions of a second.

Now imagine the ghost —CI red, local green— and the two ways to attack it. Without reproducing, your only sensor is CI: each hypothesis ("could it be this?") costs a whole pipeline round, and since you can't inspect anything, you hit it by luck. Five hypotheses are half an hour of dead waiting, plus five junk commits (fix attempt, fix attempt 2…) dirtying the history, plus the uncertainty of not knowing whether it "passed" because you nailed it or because the problem was intermittent. With reproducing, you pay a fixed cost up front —ten, fifteen minutes of setting up a clean venv with CI's versions— and in exchange you turn each following round into fractions of a second, with all your tools available (you can put a print, open the debugger, inspect values).

The math tilts very fast. Reproducing "overspends" only if the failure resolves on the first blind hypothesis —which almost never happens—. As soon as you need two or three attempts, the fixed cost of reproducing has already paid for itself, and from then on you work for free. That's why the "reproduce before fixing" rule isn't moral discipline; it's the strategy that minimizes total time. A slow pipeline makes this even truer: the more expensive the CI loop is, the more it pays to move the work to your machine, where the loop is instant. Reproducing is, at bottom, trading an expensive feedback loop (CI) for a cheap one (local) for all the fixing work.

Common mistakes

Distrusting the test before the environment. What happens: CI turns red, on your machine it passes, and your first conclusion is "the test is wrong" or "CI is slacking". You mark the test with @pytest.mark.skip, or change it to pass, and move on. Why it happens: the test is what you see turn red, so it seems like the culprit; and distrusting your own code costs more. How to spot it: if your reaction to a CI red is to touch the test without having reproduced the failure, you're about to make this mistake. How to fix it: reverse the suspicion. When the same code gives two results, suspect number one is the environment, not the test. Reproduce first; disabling a test without understanding what it protected is turning off the fire alarm because it sounds ugly.

Trying to fix without reproducing. What happens: without getting the failure to happen on your machine, you start changing lines "let's see if this fixes it" and push each attempt so CI tells you if you nailed it. Why it happens: the urgency —"we have to get the build green now"— pushes you to skip the slow step of reproducing. How to spot it: if you've done three or four pushes of "fix attempt", "fix attempt 2", "please work", you're debugging blindly through the pipeline. How to fix it: stop. Spend the time reproducing the failure locally (which is what this module teaches). A reproduced failure is fixed in a seconds-long cycle on your machine; an unreproduced one is chased for hours through CI.

Confusing reproducing with diagnosing. What happens: you get the red on your machine and, instead of celebrating it as the milestone it is, you get frustrated because "I still don't know why it fails". Why it happens: it's easy to believe reproducing and understanding are the same. How to spot it: if you expected reproducing the failure to tell you the cause, you have the two stages mixed. How to fix it: recognize that reproducing is the first victory, not the last. You already have the failure in your hand and on your turf, with all your tools. The cause is hunted now with the diagnosis techniques —from the sibling guide—, which without a reproducible failure you couldn't even start.

Exercises

Exercise 1 — Reproducing vs diagnosing. Classify each of these actions as part of reproducing (closing the environment gap) or diagnosing (understanding why the code fails). (a) Finding out which Python version the CI runner used. (b) Putting a breakpoint() inside local_start_hour to see what astimezone returns. (c) Creating a clean venv and installing the dependencies with the exact versions CI installed. (d) Reducing the test to the minimal call that triggers the error. (e) Copying the environment variables the runner had.

See solution
  • (a) Reproducing — the Python version is a piece of the environment; replicating it brings your kitchen closer to CI's.
  • (b) Diagnosing — inspecting internal values with the debugger is understanding why, and it presupposes the failure already happens on your machine.
  • (c) Reproducing — installing the exact versions is the central act of closing the dependency gap.
  • (d) Diagnosing — reducing to the minimum (minimizing the case) is a technique from the sibling guide to isolate the cause, once you already reproduce.
  • (e) Reproducing — the environment variables are part of the kitchen; copying them closes another part of the gap.

The dividing line: (a), (c), and (e) make the failure happen on your machine; (b) and (d) serve to understand it once it already happens. This module covers the first ones; the diagnosis guide, the second.

Exercise 2 — The correct suspect. A teammate writes to you: "The test test_summer_booking_starts_at_16_local has been green for months and today CI marked it red, but on my machine it's still green. I already marked it with skip to unblock the merge." Write, in two or three sentences, what you'd reply —what they did wrong and what they should do instead—.

See solution

A reasonable answer: "Marking it with skip turns off the alarm without putting out the fire: if the test protected a real rule, now that rule is uncovered and the merge goes in blind. That the same code gives green on your machine and red in CI almost always means the environment differs, not that the test is wrong —most likely here a different version of some dependency—. Instead of skipping it, reproduce the failure: look at which versions CI installed, set up a clean venv with those exact versions, and run the same command. When it comes out red for you too, we'll know what changed and whether the wrong one is the code or the test's expected number."

The key point: don't distrust the test before the environment, and don't disable a protection without understanding what it covered.

Exercise 3 — Why reproduce first. Explain in your own words, and with a timing example, why trying to fix a failure without reproducing it locally comes out more expensive than spending a while reproducing it first.

See solution

Without reproducing, your only "sensor" for whether you fixed the failure is CI itself: you change a line, commit, push, and wait for the pipeline to run —easily three to five minutes per attempt, sometimes more—. Each hypothesis costs that whole cycle, you can't put a print or inspect anything, and if you're wrong (the norm the first few times), you start over. Five blind attempts are a long half hour of waiting, plus the noise of five junk commits in the history.

Reproduced on your machine, the failure happens in seconds and with all your tools: you run pytest locally, see the red instantly, test a hypothesis, run again, seconds again. Reproducing may cost ten or fifteen minutes of preparing a clean venv with the correct versions, but after that each iteration is almost free. The math is simple: a fixed bit of setup in exchange for seconds-long cycles, versus minutes-long cycles multiplied by each attempt. And there's a benefit not measured in minutes: when you reproduce, you know what you see is the real failure; when you guess through CI, you're never sure whether it "passed" because you fixed it or because the problem was intermittent.

Summary and next step

In this lesson you gave a name to the most expensive ghost of a pipeline: CI red, your machine green (or the reverse). You saw it's not magic or a defective CI, but an environment gap: the code is identical, but the two machines differ in something —a Python version, a library version, a variable, a time zone, a file—. And you saw the rule that governs the whole module: a failure you can't reproduce you can't fix, so reproducing —closing the gap until the red appears on your machine— is the mandatory first step, before any fix attempt.

You separated two crafts people mix and that are expensive to mix: reproducing (making the failure happen on your turf, which is this whole module) and diagnosing (understanding why it fails, which is the sibling guide test-failure-diagnosis-guide). And you met the case that runs through the module: Reservo's local-time test that passes with an old pytz and fails with a new one, with the two real outputs in front of you —PASSED with 2022.1, AssertionError: assert 15 == 16 with the latest—.

Before moving on you should be able to: explain what the environment gap is; say why reproducing precedes diagnosing; distinguish a reproduction task from a diagnosis one; and argue why distrusting the test before the environment is usually the initial mistake.

What's next is opening the symptom wide. Lesson 2 makes the complete catalog of the environment gap —all the things that can differ between CI and your machine and produce the same "red here, green there"— so that when you run into one, you already have the list of suspects in your head and know where to start looking.

Resources

← Previous8. Mini-project: the `tests.yml` that runs the Reservo suiteNext →2. The symptom: red here, green there (and its causes)