7. The method for reproducing

Description

You have all the pieces: the gap catalog (lesson 2), the environment layers (3), the pinned dependencies (4), the clean venv (5), and the hidden differences (6). This lesson assembles them into a method —a repeatable, step-by-step procedure— to reproduce any CI failure on your machine. By the end you'll have a recipe to follow every time the guardian and you don't agree, instead of improvising from scratch each time.

The method has five steps, and they all point to the same thing: match your kitchen to CI's in the layers that matter, and run exactly the same command. First you read the CI log to extract the facts (which Python version, which dependency versions, which variables, which command); then you replicate each layer —the interpreter, the dependencies in a clean venv, the variables—; and finally you run the same command and watch the red appear. You're going to apply it from start to finish to Reservo's pytz failure, and take away a checklist to stick next to your monitor.

Connection to the module. This is the synthesis lesson: it turns five lessons of concepts and tools into a single procedure. It's the second-to-last because lesson 8's mini-project will have you execute this complete method on your own, from reproduction to fix. And it marks the module's boundary precisely: the method ends the instant the failure appears on your machine —from then on, the in-depth diagnosis is the sibling guide test-failure-diagnosis-guide—.

The analogy: the pilot's checklist

A pilot doesn't take off "from memory". Before each flight they go through a checklist —flaps, fuel, instruments, pressure— point by point, always in the same order, skipping none. Not because the pilot is forgetful, but because a fixed procedure eliminates human error: when you follow the list, you don't depend on your concentration that day or your intuition under pressure. The list thinks for you the steps you mustn't forget.

Reproducing a CI failure deserves the same discipline. Under the pressure of a red build, the temptation is to improvise —"let's see, I'll change this... no, better this other"— and that's how steps get forgotten (creating the clean venv, verifying the Python version, replicating the variable) and you lose track of what you tried. A fixed method, gone through always the same, turns a stressful situation into a mechanical routine: five steps, in order, and at the end you have the failure in your hand or you know exactly which layer you have left to match. The recipe thinks for you.

Said directly:

Reproducing is a procedure, not an improvisation: extract the facts from the CI log, replicate the interpreter, replicate the dependencies in a clean venv, replicate the variables, and run the same command. Followed in order, it brings the CI failure to your machine reliably.

Step 0: read the CI log to extract the facts

Before replicating anything, you need to know what to replicate. All the information lives in the CI log, and reading it with intention is half the work. There are three places to look.

The Python install step. The workflow declares the version with setup-python. In the YAML it looks like this:

# fragment of the CI workflow (.github/workflows/ci.yml)
- uses: actions/setup-python@v5
  with:
    python-version: "3.14"
- run: pip install -r requirements.txt
- run: pytest -q

That python-version: "3.14" is the first fact: CI runs Python 3.14. The pytest header in the log confirms it more precisely (Python 3.14.0).

The dependency install step, and —if the workflow prints it— a pip freeze. The command pip install -r requirements.txt tells you where the versions come from; if the workflow also runs pip freeze (a good practice, exactly for cases like this), the log lists the exact versions that were installed:

$ pip freeze          # (output in the CI log)
iniconfig==2.3.0
packaging==26.2
pluggy==1.6.0
pytest==9.1.1
pytz==2026.3.post1

There's the second fact, the most important for Reservo's failure: CI installed pytz 2026.3.post1. If your workflow doesn't print pip freeze, add it —a one-line - run: pip freeze— because without it you have to guess what the >= resolved to, and guessing is exactly what we want to avoid.

The workflow's env: block and the pytest header. The variables CI defines appear in the YAML (env:); the pytest header gives the platform (platform linux), the pytest version, the order seed if there's randomization, and the effective command. And the failure summary gives you the exact error to reproduce:

============================= test session starts ==============================
platform linux -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0
collected 1 item

test_localtime.py::test_summer_booking_starts_at_16_local FAILED          [100%]
...
FAILED test_localtime.py::test_summer_booking_starts_at_16_local - assert 15 == 16

With this you have the complete ID card of CI's environment: Python 3.14.0, pytz==2026.3.post1, pytest 9.1.1, platform linux, command pytest -q, failure assert 15 == 16. That ID card is your shopping list to replicate.

The five steps of the method

With the facts in hand, the method is this.

Step 1: replicate the Python version

Create the venv with the executable of the exact version CI used. If CI runs 3.14, use python3.14:

$ python3.14 -m venv repro-venv
$ repro-venv/bin/python --version
Python 3.14.0

Verify with --version that it matches the log's header. If you don't have that version installed, install it (with pyenv, the official installer, or whatever you use); replicating the interpreter isn't optional, because a version change can be exactly the cause. (Running the suite against several versions on purpose is module 4's matrix; here you replicate CI's.)

Step 2: replicate the dependencies in the clean venv

Install the exact versions from CI's pip freeze. The most faithful way is to save that pip freeze as a requirements-repro.txt and install it whole:

$ repro-venv/bin/python -m pip install -r requirements-repro.txt
$ repro-venv/bin/python -c "import pytz; print(pytz.__version__)"
2026.3.post1

Verify that pytz (and any suspect) ended up in CI's version. This is where lesson 4's pin and lesson 5's venv come together: you install the exact, in a clean place.

Step 3: replicate the environment variables

Match the invisible layer. If CI defines variables in its env: block, set them; if you have variables CI doesn't, remove them for the command. And if the failure smells like the system time zone, force the runner's TZ:

$ env -u RESERVO_TAX_PERCENT TZ=UTC \
    repro-venv/bin/python -m pytest ...

(In the pytz case, the variable doesn't apply; this step matters when the suspect is a variable or the TZ, as in lesson 6. But go through it always: asking yourself "which variables differ?" is part of the list.)

Step 4: replicate the directory and how your code is imported

Run from the same place as CI —the project root— so imports and relative paths resolve the same. If CI installs your project with pip install -e ., do it too:

$ cd /path/to/project        # the root, as CI's checkout does
# (if applicable) repro-venv/bin/python -m pip install -e .

Step 5: run exactly the same command and observe

Run the command identical to the log's —same flags, same target—:

$ repro-venv/bin/python -m pytest test_localtime.py -q

And observe the result. If the same red as CI appears, you reproduced the failure: end of method. If it's still green, some layer isn't matched —go back to the corresponding step (is the pytz version exactly the log's? some variable? the TZ?)— and change one thing at a time until the color moves.

Worked example: reproduce the pytz failure from start to finish

Let's apply the complete method to Reservo's failure, without skipping steps.

Step 0 — log facts: Python 3.14.0, pytz==2026.3.post1, pytest 9.1.1, command pytest -q, failure assert 15 == 16 in test_localtime.py.

Step 1 — Python:

$ python3.14 -m venv repro-venv
$ repro-venv/bin/python --version
Python 3.14.0

Step 2 — dependencies (the requirements-repro.txt is CI's pip freeze; here I install the essentials):

$ repro-venv/bin/python -m pip install pytz==2026.3.post1 pytest==9.1.1
$ repro-venv/bin/python -c "import pytz; print(pytz.__version__)"
2026.3.post1

Step 3 — variables: the failure is about the tz data (inside pytz, already pinned), not a variable or the system TZ (the test uses an explicit zone, "America/Mexico_City"). So there's no variable to match here. (Going through the step anyway confirms it's not that layer.)

Step 4 — directory: I run from the project root, where reservo/ lives.

Step 5 — the same command.

What to expect.

$ repro-venv/bin/python -m pytest test_localtime.py -q
F                                                                        [100%]
=================================== FAILURES ===================================
____________________ test_summer_booking_starts_at_16_local ____________________

>       assert local_start_hour(a_booking(), "America/Mexico_City") == 16
E       AssertionError: assert 15 == 16
E        +  where 15 = local_start_hour(Booking(id='bk-1', ...), 'America/Mexico_City')

test_localtime.py:21: AssertionError
=========================== short test summary info ============================
FAILED test_localtime.py::test_summer_booking_starts_at_16_local - assert 15 == 16
1 failed in 0.03s

Reproduced. CI's assert 15 == 16 now happens on your machine, at will. The method —five steps, in order— took you from "I don't understand, on my machine it passes" to "here's the failure, in my terminal". And notice the certainty you gained: you don't believe it's the pytz version; you know it, because you matched that layer and the failure appeared, while with your usual pytz it didn't appear.

The handoff: reproduced, now what?

Here this module ends, precisely. Reproducing the failure closed the environment gap: your kitchen and CI's now coincide in what matters, and the failure is yours, live, repeatable. But reproducing isn't fixing, and it's worth being clear about what comes next, because the next step depends on why it was failing.

• If the test's expected number aged —as in Reservo, where the 16 assumed a daylight saving time that no longer exists—, the "fix" is often to update the test (and maybe the code) to the new reality, and pin the dependency so CI and local don't diverge again. You'll do that in the mini-project (lesson 8), because it's a case where reproducing almost reveals the fix.
• If the code has a real bug that the dependency's new version uncovered, now the diagnosis begins: isolate the culprit line, use the debugger (pdb, breakpoint()), reduce the case to the minimum. That craft —understanding why the code produces the wrong value— is the sibling guide test-failure-diagnosis-guide. We don't go there; our job was to give you the reproducible failure that guide needs as a starting point.

The boundary is clean and worth keeping clear: this module takes you to having the failure in your hand; what to do with it once you have it forks between "the number/environment aged" (you close it here) and "the code has a bug" (you diagnose it there). Without the reproduction, neither branch is possible —that's why reproducing is the step that unlocks all the others—.

Deep dive: why "one layer at a time" isn't optional

The rule "change one thing at a time" appears in every reproduction lesson, and it's worth understanding why it's so inflexible, because under pressure it's very tempting to skip it. When the build is red and you want green now, the instinct is to match everything at once —new venv, Python version, all the variables, the TZ— hoping that "one of those" reproduces the failure. And sometimes it works: you reproduce. But reproducing by changing five layers at once leaves you worse off than you think, and it's worth seeing why.

The problem is that reproducing isn't the final goal; understanding what differed is. If you matched five layers together and the red appeared, you know that one of those five was the culprit, but not which. And that matters for the fix: if the cause was the pytz version, the fix is to pin pytz; if it was the TZ, the fix is to make the zone explicit in the code; if it was a variable, the fix is to document it and define it in CI. Without knowing which layer was failing, you don't know which fix to apply —you're going to "fix" all five just in case, dirtying the project with unnecessary changes, or fix the wrong one and watch the failure come back—. Changing one layer at a time isn't slowness; it's what turns "I reproduced" into "I know exactly what was wrong", which is the information the fix needs.

There's a second, subtler reason: changing everything at once can reproduce the failure for the wrong reason. Imagine the real failure was the pytz version, but when matching everything you also forced TZ=UTC, and it turns out TZ=UTC also breaks the test (by another route). You reproduce the red, yes, but now you have two causes mixed and you don't know it; you fix the pytz version, the test stays red because of the TZ, and you conclude —falsely— that the version wasn't the problem. Isolating one variable at a time is the only way to attribute the effect to its real cause, exactly like in a controlled experiment: if you move five knobs and the result changes, you learned nothing about any knob. The checklist discipline —match a layer, run, observe— is slow in appearance and very fast in practice, because each run gives you clean information instead of an ambiguous datum you'll have to untangle later.

The reproduction checklist

To stick next to the monitor. Every time you see "CI red, local green", go through it in order:

[ ] 0. Read the CI log and note the facts:
       - Python version (pytest header / setup-python)
       - dependency versions (pip freeze from the log)
       - workflow environment variables (env: block)
       - the exact pytest command (flags, target)
       - the exact failure to reproduce (assert X == Y, test name)
[ ] 1. Create a clean venv with the SAME Python version (python3.14 -m venv ...)
       -> verify with python --version
[ ] 2. Install the EXACT versions from CI's pip freeze in the clean venv
       -> verify the suspect dependency's version
[ ] 3. Match the environment variables (env -u the extras, TZ=... if applicable)
[ ] 4. Run from the project root (and pip install -e . if CI does it)
[ ] 5. Run the SAME pytest command and compare the result with CI
       -> if red same as CI: REPRODUCED (end of module)
       -> if green: match ONE more layer and repeat step 5

The checklist discipline is the pilot's: don't trust remembering the steps under pressure; go through them. And the golden rule when you don't reproduce on the first try: change one thing at a time. Never match three layers at once, because if you reproduce you won't know which it was; match one, run, observe, and that way you'll know not only that it reproduces, but what differed.

Common mistakes

Skipping step 0 and starting to replicate blindly. What happens: without reading the log, you set up a venv "with what you think CI uses" and, since you guessed the version, you don't reproduce. Why it happens: reading the log seems like a formality and there's a hurry to "do something". How to spot it: if you can't say exactly which Python and dependency version CI used, you didn't read the log. How to fix it: step 0 is the most important; without the facts, you replicate fantasies. Make sure the workflow prints pip freeze, and note the complete ID card before touching a venv.

Running a "similar" command instead of the identical one. What happens: CI runs pytest -q and you run pytest -v tests/, or the reverse, and you collect a different set of tests or in another order, and the result doesn't match. Why it happens: you use "your" usual command out of habit. How to spot it: if your command isn't character for character the log's, it's not the same. How to fix it: copy the exact command from the log —same flags, same target, same seed if there's randomization—. Reproducing is imitating, not approximating.

Declaring "it can't be reproduced" before going through all the layers. What happens: you set up a venv with CI's pytz version, it's still green, and you conclude "it's irreproducible, must be a CI thing". Why it happens: you assume the dependency was the only possible layer. How to spot it: if you gave up after matching only one or two layers, you didn't exhaust the catalog. How to fix it: go through the remaining layers —variables, TZ, directory, files, exact Python version (3.14.0 vs 3.14.1)— changing one at a time. Almost every "CI red, local green" failure is reproducible; "it can't be done" almost always means "I haven't matched the correct layer yet".

Exercises

Exercise 1 — Extract the ID card. From this CI log fragment, extract the five facts you need to reproduce:

Run actions/setup-python@v5 with python-version 3.13
...
$ pip freeze
pytz==2026.3.post1
pytest==9.1.1
...
$ TZ=UTC pytest test_localtime.py -q
platform linux -- Python 3.13.7, pytest-9.1.1
F  [100%]
FAILED test_localtime.py::test_summer_booking_starts_at_16_local - assert 15 == 16

Exercise 2 — The missing step. A teammate says: "I followed the method: I read the log, set up a venv with Python 3.14 and installed pytz==2026.3.post1. I ran pytest and... it passed. It doesn't reproduce." The CI log showed TZ=UTC before the command and the failure was assert 21 == 15. What step did they skip and how would they fix it?

$ TZ=UTC repro-venv/bin/python -m pytest test_localtime.py -q

Exercise 3 — Reproduced: now what? You reproduced two different failures. (a) The test expected 16 but with the current tz data the correct value is 15 —the number aged—. (b) The test expected 6000 of refund at 72h and now gives 5000, and reviewing you see someone changed refund_cents by mistake. For each, say whether the next step is closed in this module or belongs to the diagnosis guide, and why.

Summary and next step

In this lesson you assembled the whole module into a five-step method, preceded by step 0 —reading the CI log to extract the environment's ID card: Python version, dependency versions (from the pip freeze), variables (from the env: block), the exact command, and the failure to reproduce—. Then: (1) replicate the Python version in a venv created with that interpreter; (2) install the exact dependencies in the clean venv; (3) match the variables (env -u, TZ=...); (4) run from the root as CI's checkout does; (5) run the identical command and observe. You applied it from start to finish to Reservo's pytz failure and saw the assert 15 == 16 appear in your terminal —reproduced at will—.

You marked the handoff precisely: reproducing closes the environment gap and ends the module; what to do afterward forks between "the number/environment aged" (you close it here, as in the mini-project) and "the code has a bug" (you diagnose it in the sibling guide test-failure-diagnosis-guide). And you took away the checklist to stick next to the monitor, with its golden rule: when you don't reproduce on the first try, change one layer at a time until the color moves.

Before moving on you should be able to: recite the five steps of the method and step 0; extract the environment's ID card from a CI log; reproduce a failure following the checklist; and decide, faced with a reproduced failure, whether it's closed here or moves on to diagnosis.

What's next is putting it all to the test on your own. Lesson 8's mini-project hands you a real Reservo failure due to an unpinned dependency —CI red, local green— and asks you to reproduce it with this method, confirm it, and fix it (correct pin + updated expectation) until you leave the build green reproducibly. It's the whole module, executed from start to finish, with your hands.

Resources

• How to run pytest — pytest documentation — to copy CI's exact command (flags, target, selecting a test with ::). Reproducing is imitating the command, not approximating it.
• pip freeze — pip documentation — the command that, printed in your CI log, gives you the exact versions to replicate. Without it, step 0 becomes guesswork.
• actions/setup-python — GitHub documentation — how the workflow declares the runner's Python version (the python-version), the first fact of the ID card you replicate in step 1.
• Virtual environments (venv) — Python documentation — the tool of steps 1 and 2: creating the clean venv with the correct Python version and installing CI's exact dependencies in it.