6. Environment variables and other hidden differences

Description

The previous lesson's clean venv closes the noisiest layer of the gap —the dependencies— but leaves open the invisible layers: the differences that don't live in requirements.txt or in the code, but "in the air" of each machine. This lesson goes after them: the environment variables, the system time zone, the order in which the tests run, a file that only exists on your disk, the working directory, the locale. By the end you'll know how to recognize them, hunt them by comparing the two environments, and replicate them so your reproduction is complete when the failure wasn't (only) about a dependency.

These differences are the hardest to find precisely because they're invisible: you can read the whole project —code, tests, requirements.txt— and not find the cause, because the value that breaks the symmetry isn't in the project; it's in your shell, in the system config, on your disk. The technique changes: instead of reading the repo, you have to compare the environments. You're going to see two of these layers in action with real output —an app environment variable and the system time zone—, each moving the result of a Reservo test without touching a single line of code.

Connection to the module. Lesson 5 gave you the clean venv and warned you which layers it doesn't cover; this one deals precisely with those. It's the one that completes the arsenal before lesson 7 pulls it all together into a step-by-step method. With the dependencies (lessons 4-5) and the hidden differences (this one) under control, you'll have covered the whole gap that a "CI red, local green" failure usually hides. On the Reservo suite.

The analogy: what the recipe takes for granted

Go back to the recipe that only turns out for you. You already ruled out the written ingredients (the "dependencies": same flour, same brand). And still, it comes out different for you. The cause is in what the recipe takes for granted and doesn't write: "water boils at 100°" (but you live at 2000 meters, where it boils at 93°), "at room temperature" (but your kitchen is at 15° and your friend's at 28°), "use your usual salt" (but yours is coarser). None of that is in the ingredient list; they're conditions of the environment the recipe assumes identical and that aren't.

The environment's hidden differences are that: conditions your code takes for granted without declaring them. "The variable RESERVO_TAX_PERCENT will be set" (but in CI it isn't). "The machine is on Mexico City time" (but the runner is on UTC). "The file sample_bookings.csv will be there" (but it wasn't committed). The code doesn't mention them because, in the kitchen where it was written, they were always present —just like your kitchen's altitude, which you never note down because to you it's constant—. Hunting them is asking yourself what my code is assuming that isn't met in the other kitchen, and to answer that you have to look at the environment, not the recipe.

Said directly:

The hidden differences are environment conditions your code takes for granted without declaring them: variables, time zone, files, directory, locale. They're not seen by reading the project —because they're not in it— but by comparing the two environments. The cure is to make them explicit.

The catalog of the invisible

Let's review the invisible layers, with their mechanism and how they're hunted.

Environment variables

Values your code reads with os.environ and that live in the shell, not in the project. Configuration, flags, credentials, rates. Your shell can have dozens accumulated; CI starts with a minimal set. How to hunt them: env or printenv list all of your shell's; compare that list with the ones the CI workflow defines (in its env: block). What you have and CI doesn't —or with another value— is a suspect. How to replicate them: to reproduce CI's environment, run the test without the variables CI doesn't have (env -u NAME pytest) or with the values CI uses (NAME=value pytest).

The system time zone

The TZ variable (and the operating system's time-zone configuration) determines what "local time" is for code that doesn't specify an explicit zone. Your laptop is in your city; the runner is almost always in UTC. A datetime that's converted to "local" without saying to which falls at different hours. How to hunt it: echo $TZ, printenv TZ, or date (shows the system's time and zone). How to replicate it: force the same one with TZ=UTC pytest to imitate the runner. (Careful: this is different from the tz data, which travels in pytz and is controlled by pinning the dependency —lesson 4—. Here we're talking about which zone the system uses, not what it knows about that zone.)

The order of the tests

pytest collects and runs the tests in some order; plugins like pytest-randomly shuffle it with a seed. If two tests share state by accident, the result depends on who ran before. CI and your machine can order differently (because of the file system, because of the seed). How to hunt it: if the culprit test passes when you run it alone (pytest test_x.py::test_culprit) but fails in the complete suite, there's order coupling. The pytest-randomly seed appears in the log header (Using --randomly-seed=...). How to replicate it: run with the same seed (pytest -p randomly --randomly-seed=<CI's>) to reproduce the same order. The diagnosis of why the tests are coupled —and how to decouple them— is from the sibling guide test-failure-diagnosis-guide; here it's enough to replicate the order to reproduce.

A file that only exists on your disk

Your test reads a file —a .env, a data CSV, a fixture— that's on your machine but not in the repository (because of .gitignore or forgetting to commit). On your disk it exists; on the runner, which only has git, it doesn't. How to hunt it: the CI failure is usually FileNotFoundError; the file it mentions exists on your disk but git ls-files doesn't list it. How to replicate it: to reproduce the absence, temporarily rename the file on your disk and run; or better, clone the clean repo in another folder (which will only have what's committed) and run there. Related: hardcoded absolute paths (/Users/yourname/...) that only exist on your machine.

The working directory and the locale

Two more, cross-cutting. The working directory: where you run pytest from changes how imports and relative paths resolve (running from the root vs from tests/). It's hunted with pwd and with the rootdir pytest prints; it's replicated by running from the same place as CI (the root). The locale (LANG, LC_ALL): it affects how text is sorted, how numbers and dates are formatted, which encoding is assumed. Your machine can be on es_MX.UTF-8 and the runner on C or C.UTF-8. It's hunted with env | grep -E 'LANG|LC_' and replicated by exporting the same value.

Worked example 1: an app environment variable

Let's take up Reservo's total_with_tax_cents again, which reads the tax rate from an environment variable:

# reservo/config.py
import os


def tax_percent():
    return int(os.environ.get("RESERVO_TAX_PERCENT", "0"))


def total_with_tax_cents(price_cents):
    return price_cents + price_cents * tax_percent() // 100

# test_config.py
from reservo.config import total_with_tax_cents


def test_pro_3h_total_with_tax():
    assert total_with_tax_cents(6000) == 6960   # expects 16% tax

The dev has RESERVO_TAX_PERCENT=16 exported in their shell; CI doesn't have it. To reproduce CI's failure on your machine, the key isn't a clean venv (which doesn't erase shell variables), but running the test without the variable, imitating the runner's minimal environment. And to confirm that variable is the cause, you run the two versions and compare.

What to expect. With the variable set (replicates the dev's machine):

$ RESERVO_TAX_PERCENT=16 python -m pytest test_config.py -q
.                                                                        [100%]
1 passed in 0.00s

Without the variable (replicates CI's clean runner):

$ env -u RESERVO_TAX_PERCENT python -m pytest test_config.py -q
F                                                                        [100%]
=================================== FAILURES ===================================
_________________________ test_pro_3h_total_with_tax __________________________

>       assert total_with_tax_cents(6000) == 6960
E       assert 6000 == 6960
E        +  where 6000 = total_with_tax_cents(6000)

test_config.py:6: AssertionError
=========================== short test summary info ============================
FAILED test_config.py::test_pro_3h_total_with_tax - assert 6000 == 6960
1 failed in 0.02s

The env -u RESERVO_TAX_PERCENT runs the command removing that variable —it's your way of imitating CI's clean shell without having to log out or touch your .zshrc—. You reproduced the red: assert 6000 == 6960, the same one CI saw. And along the way you confirmed the cause: with the variable, green; without it, red; the variable is the difference. Notice here the clean venv wouldn't have been enough —the failure wasn't about a dependency—; the correct tool was controlling the variable.

Worked example 2: the system time zone

Reservo also has a little function that shows a booking's time according to the machine's system clock —without specifying an explicit zone, trusting the environment's TZ—:

# reservo/localclock.py
def utc_to_system_local_hour(dt_utc):
    """Wall-clock hour on the system clock (depends on the environment's TZ)."""
    return dt_utc.astimezone().hour   # astimezone() with no argument uses the system TZ

# test_localclock.py
from datetime import datetime, timezone
from reservo.localclock import utc_to_system_local_hour

BOOKING_UTC = datetime(2026, 3, 10, 21, 0, tzinfo=timezone.utc)   # 21:00 UTC


def test_booking_shows_at_15_on_the_wall_clock():
    # The dev's machine is in America/Mexico_City (UTC-6): 21:00 UTC = 15:00.
    assert utc_to_system_local_hour(BOOKING_UTC) == 15

The dev wrote == 15 because their laptop is in Mexico City (UTC−6). The CI runner is in UTC, where 21:00 UTC is 21:00 "local". Same function, same input, different system TZ.

What to expect. With the dev's TZ:

$ TZ=America/Mexico_City python -m pytest test_localclock.py -q
.                                                                        [100%]
1 passed in 0.00s

With the runner's TZ:

$ TZ=UTC python -m pytest test_localclock.py -q
F                                                                        [100%]
=================================== FAILURES ===================================
________________ test_booking_shows_at_15_on_the_wall_clock ________________

>       assert utc_to_system_local_hour(BOOKING_UTC) == 15
E       AssertionError: assert 21 == 15
E        +  where 21 = utc_to_system_local_hour(datetime.datetime(2026, 3, 10, 21, 0, tzinfo=datetime.timezone.utc))

test_localclock.py:10: AssertionError
=========================== short test summary info ============================
FAILED test_localclock.py::test_booking_shows_at_15_on_the_wall_clock - assert 21 == 15
1 failed in 0.02s

TZ=UTC reproduces the runner's red: assert 21 == 15. You reproduced the failure by forcing the same system time zone as CI, without moving anything else. Notice the difference from the pytz failure of the previous lessons: that one was the time-zone data (what the library knows about daylight saving), and it was controlled by pinning the dependency; this is the system zone (which city the machine "thinks" it's in), and it's controlled with the TZ variable. Two different layers, two different tools, the same surface symptom —a shifted hour—. That's why the catalog matters: without it, you'd confuse one with the other and look in the wrong place.

How to hunt a hidden difference: compare, don't read

The general method for these invisible layers isn't to read the code —we already saw the cause isn't there— but to put the two environments side by side and look for what differs. A concrete procedure:

1. Extract CI's environment from the log. The workflow declares its Python version, its variables (env: block), and sometimes prints useful things. The pytest header gives the platform, the Python and pytest version, and the order seed if there's randomization.
2. Photograph your environment. python --version, pip freeze, env, echo $TZ, pwd, git status. It's the list of everything your kitchen has.
3. Compare layer by layer. Does the Python version match? Do the pip freeze versions match? What variables do you have that the workflow doesn't define? Is your TZ the same as the runner's (UTC)? Do you run from the same directory? Are there files you have and git ls-files doesn't?
4. Replicate the suspect difference and run. Change one thing (remove a variable with env -u, force TZ=UTC, run from the root) and observe whether the result changes. Changing one at a time tells you which layer was the culprit.

This "compare and change one at a time" is the reproduction version of a scientific method: one hypothesis at a time, one change at a time, to attribute the effect to the correct cause. It's slow compared to "read the code and guess", but it's sure, and with a handful of comparisons you close the gap that a month of rereading the repo wouldn't have closed.

Common mistakes

Searching the code for a cause that lives in the shell. What happens: the failure is due to a variable or the TZ, but you reread config.py and localclock.py over and over looking for the bug. Why it happens: it's natural to search inside the project, where you have control. How to spot it: if the code is identical on both machines and they still differ, the cause isn't in the code. How to fix it: stop reading the repo and compare the environments (env, printenv, echo $TZ, pip freeze). The difference you're looking for isn't in the recipe; it's in the environment.

Trusting that the clean venv catches everything. What happens: you set up a perfect venv with CI's versions, the test keeps passing (you don't reproduce), and you're baffled. Why it happens: you believe the venv isolates the whole environment. How to spot it: if the failure depends on a variable or the TZ, the venv —which doesn't erase your shell's variables— won't catch it. How to fix it: remember which layers the venv covers (dependencies, interpreter) and which it doesn't (variables, system tz, files). For those, control the variable directly: env -u, TZ=..., running from another folder.

Changing several things at once when reproducing. What happens: to "make sure", you set up a new venv and remove three variables and change the TZ, all at once; you reproduce the red but don't know which of the changes caused it. Why it happens: the hurry to reproduce pushes you to change everything at once. How to spot it: if you reproduced but can't say which layer was the culprit, you changed too much at once. How to fix it: change one thing at a time. Set the hypothesis (for example, "it's the TZ"), change only that, run; if it reproduces, that was it; if not, revert and try the next. One change at a time is what turns reproducing into understanding what differed.

Exercises

Exercise 1 — Choose the tool. For each CI failure (green on your machine), say whether a clean venv would be enough to reproduce it, and if not, with which command you'd reproduce it. (a) CI installed pytz 2026.3.post1 and you have 2022.1. (b) Your code reads RESERVO_TAX_PERCENT, which you have exported and CI doesn't. (c) A function uses datetime.astimezone() with no zone, your machine is in CDMX and the runner in UTC. (d) A test reads sample.csv, which exists on your disk but not in the repo.

Exercise 2 — Hunt by comparison. A test passes on your machine and fails in CI with assert 21 == 15 in a time calculation. You have the same Python, the same pip freeze as the CI log, and the same directory. What would you compare next, with which command, and what would you expect to find?

$ echo $TZ            # or: printenv TZ
$ date                # shows the system's time and zone

$ TZ=UTC python -m pytest test_localclock.py -q     # should reproduce the red

Exercise 3 — Make it explicit. The test test_booking_shows_at_15_on_the_wall_clock depends on the system TZ, which is why it passes locally and fails in CI. Beyond reproducing it, what would you change so the test doesn't depend on an invisible layer and gives the same result on any machine? (Hint: make the implicit explicit.)

Summary and next step

In this lesson you hunted the hidden differences: the environment layers the clean venv doesn't cover because they don't live in requirements.txt but "in the air" of each machine. You went through the catalog of the invisible —environment variables, system time zone (TZ), test order, a file that only exists on your disk, working directory, locale— with their mechanism, how to hunt them (env, printenv, echo $TZ, git ls-files) and how to replicate them (env -u, TZ=..., running from the root). You saw two in action with real output: an app variable (RESERVO_TAX_PERCENT, which moves the total from 6960 to 6000) and the system time zone (TZ, which moves the hour from 15 to 21), each reproducible by controlling the correct layer.

And you learned the method for these layers: compare, don't read. The cause isn't in the repo (which is identical on both machines), so it's hunted by putting the two environments side by side and changing one thing at a time until the result moves. The underlying cure, beyond reproducing, is to make the implicit explicit: declare in the code or the test every environment condition the result depends on.

Before moving on you should be able to: name the invisible layers and how they're hunted; decide when a venv is enough and when you have to control a variable or the TZ; reproduce a variable failure with env -u and a zone one with TZ=; and change one thing at a time to attribute the correct cause.

What's next is pulling the whole arsenal —the CI log, the Python version, the pinned dependencies, the clean venv, the variables and the zone— together into a repeatable, step-by-step method to reproduce any CI failure. Lesson 7 assembles that complete procedure and applies it from start to finish to Reservo's pytz failure, so you have a recipe to follow every time the guardian and you don't agree.

Resources

• os.environ — Python documentation — the interface through which the code reads environment variables, the invisible layer par excellence. Understanding that it reflects the state of each machine explains why a variable breaks the symmetry.
• zoneinfo module — Python documentation — the modern way to handle explicit time zones in the standard library. It's the tool for "making the zone explicit" and stop depending on the system TZ.
• Environment variables in GitHub Actions — GitHub documentation — how a workflow defines (or not) its variables, and what a clean runner starts with. It's the "other environment" you compare against yours.
• How to run pytest — pytest documentation — to replicate CI's exact order and options, including selecting a single test (::) that helps hunt order coupling.