Module 3: Reproducing A Ci Failure Locally

3. The environment gap, up close

Next →

Description

We've been talking about "the environment" as if we all knew what it is, but it's time to open it up and look at it piece by piece. By the end of this lesson you'll have a precise mental model of what makes up your tests' execution environment —Python, dependencies, variables, operating system, time zone, file system, working directory— and you'll understand why the CI runner and your machine are, by their very nature, two different worlds that only coincide if you make the effort to make them coincide.

The central idea is an image that will stick with you: CI is a kitchen that's set up and torn down on every run —clean, empty, ephemeral—; your machine is a kitchen where you've been cooking for years, with sediment in every corner. That asymmetry isn't a defect of either; it's what they are. Reproducing a failure is, at bottom, setting up on your machine a kitchen as clean and controlled as CI's. To achieve that you first have to know what a kitchen is made of, and that's what this lesson does.

Connection to the module. Lesson 2 gave you the catalog of suspects. This one gives you the map of the ground where they live: what the environment is, layer by layer, and why it differs between the two machines. It's the conceptual base the concrete solutions stand on: lesson 4 (pinning dependencies) attacks the library layer; lesson 5 (the clean venv) attacks the accumulated sediment; lesson 6 (variables and hidden differences) attacks the invisible layers. Here you'll also see, with real output, a hidden difference in action —an environment variable that moves the result of a Reservo test—.

The analogy: two kitchens

Imagine two kitchens where the same recipe is going to be prepared.

The CI kitchen is a TV set's: they set it up from scratch for each recording. They bring a new stove, freshly bought ingredients dated today, clean utensils, and when they finish they tear it all down. Nothing is left over from the previous recording. It's reproducible by design: if tomorrow they set up the same kitchen with the same shopping list, it comes out identical. But precisely because of that it's strict: if your recipe needed a pinch of something that "was always in the pantry", in this kitchen it isn't there, because the pantry is empty except for what you asked for explicitly.

Your home kitchen is where you cook every day and have for years. It has a stove whose quirks you know, spices opened months ago, a jar of yeast you bought for another recipe, salt in a shaker you refill without thinking. Everything works because you know where each thing is and what's available. But none of that is written on any list. If you gave the recipe to someone with an empty kitchen, they'd be missing ten things that are invisible to you because "they're always there".

The environment gap is the distance between these two kitchens. Your recipe turns out for you because your kitchen has, without you noticing, everything it needs. In CI's clean kitchen, anything you took for granted and didn't write on the shopping list —a library you installed a year ago, a variable you exported once, a file you left on the desk— simply isn't there. Reproducing the CI failure is cooking in a kitchen as empty as theirs, so that you too are missing what they're missing. And for that you have to know what layers make up a kitchen.

Said directly:

The execution environment is several layers —interpreter, dependencies, variables, system, time zone, files, directory—. CI sets up those layers clean and explicit on every run; your machine has them sedimented and implicit. The gap is everything your machine has "extra" without it being written anywhere.

The layers of the environment

Let's go layer by layer. Think of each one as a stratum that can differ between the two machines.

The Python interpreter

The deepest layer: which python runs your tests, and of what version. It's not just "3.14 vs 3.13"; it's also which 3.14 —the system's, a pyenv's, a venv's—. Each version brings its standard library, its syntax, and its behaviors. CI declares its version explicitly in the workflow (setup-python with python-version: "3.14"); your machine runs the python you have on the PATH, which can be any. How to see it: python --version.

The installed dependencies

The third-party library layer: pytz, requests, pytest, and everything your project imports that isn't from the standard library. Not only which are installed, but in what exact version. This is the layer where the queen of the module's causes lives. CI installs these dependencies fresh on every run, from your requirements.txt; your machine has them from whenever you installed them, with that day's versions. How to see it: pip list or pip freeze.

The environment variables

The invisible layer: values that live in the shell and that your code reads with os.environ. Configuration, credentials, flags, the default time zone (TZ), the locale (LANG, LC_ALL). They're not in any project file; they're "in the air" of each machine. Your shell can have dozens you don't even remember setting; CI starts with a minimal and clean set. How to see it: env or printenv.

The operating system

The layer of the machine itself: Linux, macOS, or Windows; and within Linux, which distribution. The line endings change (\n vs \r\n), the path separators (/ vs \), the case sensitivity of file names (Linux distinguishes Data.csv from data.csv; macOS by default doesn't), which system commands exist. CI almost always runs Linux (ubuntu-latest); your machine is usually macOS or Windows. You saw it in the log header: platform linux in CI, platform darwin on your Mac. How to see it: uname -a (or the pytest header).

The time zone and its data

A sub-layer between the system and the dependencies: the machine's current time zone (America/Mexico_City vs UTC) and the installed time-zone database (which says, for each zone, when there was daylight saving time). The first is the system's; the second travels inside libraries like pytz or the system's data (tzdata), and gets updated over time. It's the layer that breaks Reservo's test. How to see it: the TZ variable, and the version of pytz/tzdata.

The file system

The layer of "which files exist and where": the repository's (which CI has because it did checkout), plus anything your machine has and the repo doesn't —a .env, uncommitted test data, a file you created by hand—. CI only has what's in git plus what the workflow creates explicitly; your machine has everything that has ever passed through your disk. How to see it: git status, git ls-files, and comparing against what your code reads.

The working directory

The subtlest layer: where pytest is run from. Imports and relative paths resolve differently depending on the current directory. CI runs from the repo root (the checkout leaves the project there); you can run from wherever you're standing in the terminal. Running pytest from the root or from tests/ can change what gets imported and which files are found. How to see it: pwd, and which rootdir pytest starts with (it prints it in the header).

Why CI is clean and your machine isn't

The underlying difference between the two kitchens isn't configuration, it's lifecycle. The CI runner is ephemeral: it's born when your run starts and dies when it finishes. Every push gets a freshly created virtual machine (or a container), with no memory of anything before. That has an enormous consequence: in CI, only what the workflow installs or creates explicitly exists. If your code needs pytz, it has to be in requirements.txt, because there's no pytz "from before". If it needs a variable, the workflow has to set it, because the shell starts almost empty. That forced discipline is uncomfortable at first, but it's exactly what makes CI reproducible: since it doesn't depend on any sediment, two identical runs give the same.

Your machine is the opposite: persistent and cumulative. You've spent months or years installing packages, exporting variables, leaving files, testing things. Each of those actions left sediment, and your project can be using that sediment without you knowing. You installed pytz a year ago for another project and it stayed there; your test uses it and it works, but it's not in your requirements.txt because "it was already there". You exported RESERVO_TAX_PERCENT in a session and it stayed in your .zshrc; your test reads it and passes, but the runner doesn't have it. That sediment is comfortable —things "just work"— and that's why it's dangerous: it hides your project's real dependencies from you, until CI, with its empty kitchen, reveals them to you one by one in the form of red.

This reframes what reproducing a failure really is. It's not "making my machine behave strangely"; it's taking away from my machine the sediment that makes it too kind, until it resembles CI's clean kitchen. That's why the central tool of reproduction (lesson 5) is a clean virtual environment: a new, empty corner inside your machine where only what you install on purpose exists, just like in CI.

Worked example: a hidden difference, with real output

Let's make an invisible layer visible. Reservo has a little function that computes a booking's total by adding a local tax, and it reads that tax percentage from an environment variable (so it can be configured per country without touching the code):

# reservo/config.py
import os


def tax_percent():
    """Reservo reads the local tax rate from the environment (default 0)."""
    return int(os.environ.get("RESERVO_TAX_PERCENT", "0"))


def total_with_tax_cents(price_cents):
    return price_cents + price_cents * tax_percent() // 100

And its test, which a dev wrote on their machine —where they'd had RESERVO_TAX_PERCENT=16 exported for a while—:

# test_config.py
from reservo.config import total_with_tax_cents


def test_pro_3h_total_with_tax():
    # pro 3h = 6000; the dev has RESERVO_TAX_PERCENT=16 exported in their shell.
    assert total_with_tax_cents(6000) == 6960

The expected number comes from the math: 6000 cents (what a pro pays for 3 hours of Focus) plus 16% tax (6000 * 16 // 100 = 960) gives 6960. On the dev's machine, with the variable set, it passes. On CI's clean runner, without the variable, the tax is 0 and the total is 6000, so the assert 6000 == 6960 fails.

I ran it for real in a single environment, changing only whether the variable is set or not —to isolate that single layer—.

What to expect. With the variable exported (the dev's machine):

$ RESERVO_TAX_PERCENT=16 python -m pytest test_config.py -q
.                                                                        [100%]
1 passed in 0.00s

And without the variable (CI's clean runner):

$ env -u RESERVO_TAX_PERCENT python -m pytest test_config.py -q
F                                                                        [100%]
=================================== FAILURES ===================================
_________________________ test_pro_3h_total_with_tax __________________________

    def test_pro_3h_total_with_tax():
        # pro 3h = 6000; the dev has RESERVO_TAX_PERCENT=16 exported in their shell.
>       assert total_with_tax_cents(6000) == 6960
E       assert 6000 == 6960
E        +  where 6000 = total_with_tax_cents(6000)

test_config.py:6: AssertionError
=========================== short test summary info ============================
FAILED test_config.py::test_pro_3h_total_with_tax - assert 6000 == 6960
1 failed in 0.02s

Same code, same test, same version of everything. The only difference is an invisible layer of the environment —an environment variable— and with it the result jumps from 6960 to 6000, and the color from green to red. Notice how treacherous it is: if you read the whole project's code, you'll never find why the two machines disagree, because the value that differentiates them (RESERVO_TAX_PERCENT=16) isn't in the project —it's in the dev's shell—. This is why the invisible layers (variables, tz, system) are the hardest to hunt: they're not seen by reading the repo; they're only seen by comparing the two environments. How to do that comparison is lesson 6.

Deep dive: making the implicit explicit

If the environment gap is "everything your machine has extra without it being written", then the underlying cure —beyond reproducing a one-off failure— is a discipline: making the implicit explicit. Everything your project depends on should be declared somewhere in the repository, not live as sediment on your machine.

  • Dependencies go in requirements.txt (lesson 4), with exact versions, so the library layer is identical in any kitchen.
  • The Python version goes declared —in the CI workflow, and often in a file like .python-version or in the pyproject.toml— so the interpreter layer isn't left to chance.
  • The environment variables the project needs go documented (a .env.example that is committed, even if the real .env isn't), and the CI workflow defines them explicitly, so the invisible layer stops being invisible.
  • The data files the tests need go committed (or generated by the test itself), so the file-system layer doesn't depend on your disk.

When everything the project needs is declared, the environment gap shrinks until it almost disappears: CI's kitchen and a clean kitchen on your machine have the same shopping list, so they come out the same. Reproducing a failure, then, stops being an archaeology ("what does my machine have that CI's doesn't?") and becomes mechanical ("I install exactly what's declared, in a clean environment, and run the same command"). That's the destination the module points at, and this lesson is the map that makes it possible: you can't close a gap whose layers you don't know.

Common mistakes

Believing "the environment" is just the Python version. What happens: when something differs between CI and local, you only check python --version and, if it matches, you conclude "the environments are the same". Why it happens: the Python version is the best-known and easiest-to-compare layer. How to spot it: if your environment checklist has a single item, you're missing six. How to fix it: the environment is all the layers —interpreter, dependencies, variables, system, tz, files, directory—. Python matching says nothing about whether pytz matches, or whether you have a variable the runner doesn't.

Trusting that "it was already installed" is enough. What happens: your project imports a library that isn't in requirements.txt, but it works on your machine because you installed it a while ago for something else. Why it happens: your machine's sediment makes everything "just work", and you don't notice the undeclared dependency. How to spot it: if your code does import X and X doesn't appear in requirements.txt, you have an implicit dependency waiting to explode in CI's clean kitchen. How to fix it: declare every dependency you import. The acid test is lesson 5: in a clean venv, import X will fail with ModuleNotFoundError if it's not declared.

Searching the repository for a difference that lives in the shell. What happens: a failure is due to an environment variable or the system time zone, but you reread the project's code over and over looking for the cause. Why it happens: it's natural to look for the difference "inside the project", where you have control. How to spot it: if you've spent a while reading the repo and found nothing that differs —because the repo is identical on both machines—, the difference probably lives in a layer that isn't in the repo. How to fix it: stop reading the code and compare the environments: env/printenv for variables, pip freeze for dependencies, python --version for the interpreter. The difference you're looking for isn't in the project; it's in the kitchen.

Exercises

Exercise 1 — Locate the layer. For each difference, say which layer of the environment it lives in and with which command you'd see it. (a) CI has requests 2.32 and you have requests 2.28. (b) CI runs Linux and you macOS. (c) Your shell has LANG=es_MX.UTF-8 and the runner LANG=C. (d) A test CSV exists on your disk but not in the repo. (e) You run pytest from tests/ and CI from the root.

See solution
  • (a) Installed dependencies layer; seen with pip freeze (or pip show requests).
  • (b) Operating system layer; seen with uname -a or the pytest header (platform linux vs platform darwin).
  • (c) Environment variables layer (the locale); seen with env | grep LANG or printenv LANG.
  • (d) File system layer; seen with git status / git ls-files (the file doesn't appear if it's not committed).
  • (e) Working directory layer; seen with pwd and with the rootdir pytest prints in its header.

Exercise 2 — Predict the result. Given the lesson's total_with_tax_cents, what would python -m pytest test_config.py -q print in each case? (a) RESERVO_TAX_PERCENT=0. (b) RESERVO_TAX_PERCENT=10. (c) The variable isn't set. Justify each number.

See solution

The test expects total_with_tax_cents(6000) == 6960, which corresponds to a 16% tax (6000 + 6000*16//100 = 6000 + 960 = 6960).

  • (a) RESERVO_TAX_PERCENT=0: the total is 6000 + 6000*0//100 = 6000. assert 6000 == 6960 fails → 1 failed.
  • (b) RESERVO_TAX_PERCENT=10: the total is 6000 + 6000*10//100 = 6600. assert 6600 == 6960 fails → 1 failed.
  • (c) The variable isn't set: os.environ.get(..., "0") returns "0", so the tax is 0 and the total is 6000. fails → 1 failed (identical to case a).

It would only pass with RESERVO_TAX_PERCENT=16. The test is tied to a value of an invisible layer; that coupling is the problem. (A more robust test would set the value explicitly in the test —with monkeypatch.setenv— instead of depending on the shell; but that's test design, a topic for the sibling guides.)

Exercise 3 — The shopping list. Your Reservo project imports pytz and reads the variable RESERVO_TAX_PERCENT, and its tests load a sample_bookings.csv. List what you'd have to declare/commit —and where— so that a clean kitchen (CI or a new venv) has everything the project needs, without depending on your machine's sediment.

See solution

So that any clean kitchen reproduces your project:

  1. pytz → declare it in requirements.txt with an exact version (pytz==2026.3.post1), so the dependency layer is identical. Without this, a clean venv will give ModuleNotFoundError: No module named 'pytz'.
  2. The Python version → declare it in the CI workflow (setup-python with python-version) and, optionally, in a .python-version/pyproject.toml, so the interpreter layer isn't left to chance.
  3. RESERVO_TAX_PERCENT → document that it exists (for example, a committed .env.example with RESERVO_TAX_PERCENT=16) and define it explicitly in the CI workflow, so the invisible layer stops being invisible. Ideally, also, the tests shouldn't depend on the shell: they should set the value themselves.
  4. sample_bookings.csv → commit it to the repository (or have the test generate it), so the file-system layer doesn't depend on your disk. And use paths relative to the project, not absolute ones from your machine.

With those four things declared, the project's shopping list is complete: an empty kitchen can set up exactly what's needed and reproduce your result. Everything left undeclared is a future red surprise in CI.

Summary and next step

In this lesson you opened up "the environment" and looked at it inside: its layers —Python interpreter, installed dependencies, environment variables, operating system, time zone and its data, file system, working directory— and the command with which you'd see each one. And you understood the underlying asymmetry with the two-kitchens image: CI is ephemeral and clean (only what it installs explicitly exists, which is why it's reproducible), your machine is persistent and cumulative (full of sediment that makes everything "just work", which is why it hides your real dependencies). Reproducing a failure is taking that sediment away from your machine until it resembles CI's kitchen.

You saw an invisible layer in action, with real output: an environment variable (RESERVO_TAX_PERCENT) that moves a booking's total from 6960 to 6000 and the color from green to red, without anything in the project explaining the difference —because the value that causes it lives in the shell, not in the repo—. And you drew the underlying discipline: make the implicit explicit, declare in the repository everything the project depends on, so the gap shrinks until it almost disappears.

Before moving on you should be able to: name the seven layers of the environment and how each is inspected; explain why CI is clean and your machine isn't, in terms of lifecycle; and argue why the invisible layers (variables, tz) aren't hunted by reading the repo but by comparing environments.

What's next is going down to the noisiest layer of all —the dependencies— and to the tool that tames it. Lesson 4 dissects requirements.txt, the difference between pinning with == and leaving a range with >=, and why that detail is what makes your local and CI end up with different versions of a library. It's the root cause of Reservo's failure, and you're going to see it happen with a real venv.

Resources

← Previous2. The symptom: red here, green there (and its causes)Next →4. Pinned dependencies vs ranges (`==` vs `>=`)