2. pytest in GitHub Actions

Overview

You have tests. You run them locally when you remember. Sometimes they pass, sometimes they fail, and sometimes you simply don't run them because "I only changed one line." This is the problem CI solves: taking the tests off your machine and moving them to a neutral environment that runs them automatically on every push, no excuses, no forgetting.

In this capsule you connect pytest with GitHub Actions. It isn't complicated, but there are details that matter: how to structure the workflow, which pytest flags to use in CI vs locally, how to read the logs when something fails, and how to handle the differences between your local environment and the runner. When you finish, every push to your repo will run your tests automatically and you'll see the result directly on GitHub.

The core idea is simple: if a test passes on your machine but fails in CI, it doesn't pass. The CI runner is the neutral judge. Your machine has configurations, environment variables, and specific versions that can hide bugs. The runner starts clean every time — that's its advantage.

The complete workflow: From checkout to pytest

The pattern for running pytest in GitHub Actions has 4 steps that always repeat in the same order:

1. Checkout     → Bring your code to the runner
2. Setup Python → Install the Python version you need
3. Install deps → Install the project's dependencies
4. Run pytest   → Run the tests

Let's look at the complete workflow:

# .github/workflows/test.yml
name: Tests
# Both triggers needed: push catches broken main, pull_request validates before merge
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    # ubuntu-latest is cheapest and fastest — use macOS/Windows only if your app requires them
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run tests
        # -v shows each test name — critical in CI where you can't re-run interactively
        run: pytest tests/ -v

Why this order?

Each step depends on the previous one:

If you change the order, the workflow fails. This order is so standard that you'll see it in practically every Python repo with CI.

Setting up the project for CI

Before configuring the workflow, your project needs a minimal structure:

my-ai-project/
├── .github/
│   └── workflows/
│       └── test.yml
├── src/
│   └── ai_app/
│       ├── __init__.py
│       ├── chain.py
│       └── utils.py
├── tests/
│   ├── __init__.py
│   ├── conftest.py
│   ├── test_chain.py
│   └── test_utils.py
├── requirements.txt
└── pyproject.toml

requirements.txt for CI

Your requirements.txt should include the testing dependencies:

# requirements.txt
langchain==0.3.14
openai==1.58.1
pydantic==2.10.4
pytest==8.3.4
pytest-cov==6.0.0

# requirements.txt (production)
langchain==0.3.14
openai==1.58.1
pydantic==2.10.4

# requirements-dev.txt (testing)
-r requirements.txt
pytest==8.3.4
pytest-cov==6.0.0
pytest-timeout==2.3.1

      - name: Install dependencies
        run: pip install -r requirements-dev.txt

pytest flags for CI

When you run pytest locally, the default is fine. In CI you need more information and more control.

-v (verbose)

# Without -v
tests/test_chain.py ..F.

# With -v
tests/test_chain.py::test_chain_initializes PASSED
tests/test_chain.py::test_chain_processes_input PASSED
tests/test_chain.py::test_chain_handles_empty_input FAILED
tests/test_chain.py::test_chain_returns_structured_output PASSED

In CI you always want -v. When a test fails, you need to know exactly which one it was.

--tb=short (short traceback)

# With --tb=long (default) — 20+ lines of traceback
# With --tb=short — one line with the concrete error:
FAILED tests/test_chain.py::test_chain_handles_empty_input - ValueError: Input cannot be empty

--tb=short reduces noise in the CI logs. You see what failed and why, without excessive traceback.

-x (stop on first failure)

pytest tests/ -v -x

Stops at the first test that fails. Useful when you have 200 tests and the first one already failed.

Recommended combination for CI

      - name: Run tests
        run: pytest tests/ -v --tb=short

If you have a large test suite and want fast feedback:

      - name: Run tests (fail fast)
        run: pytest tests/ -v --tb=short -x

Comparison table of flags

Reading pytest output in Actions logs

When a test fails in CI, you need to read the logs in GitHub Actions' UI:

Run pytest tests/ -v --tb=short
============================= test session starts ==============================
platform linux -- Python 3.12.8, pytest-8.3.4, pluggy-1.5.0
collected 12 items

tests/test_utils.py::test_format_prompt PASSED                          [  8%]
tests/test_utils.py::test_validate_input PASSED                         [ 16%]
tests/test_chain.py::test_chain_processes_input FAILED                  [ 41%]

FAILED tests/test_chain.py::test_chain_processes_input - AssertionError: assert 'error' == 'success'
=========================== short test summary info ============================
FAILED tests/test_chain.py::test_chain_processes_input
========================= 1 failed, 4 passed in 0.28s =========================

What to look for in the logs

1. collected N items — Were all the tests found? If it says 0 items, pytest didn't find your tests.
2. FAILED — Which specific test failed.
3. The error message — After FAILED, the line with - gives you the concrete error.
4. passed in Xs — The total time. If it's suspiciously long, there may be slow tests.

PYTHONPATH: The most common problem in CI

This is the most frequent error when you move tests from local to CI:

ModuleNotFoundError: No module named 'src.ai_app'

Why does it happen?

Locally, your IDE configures the PYTHONPATH automatically. The CI runner doesn't have that configuration. When your test does from src.ai_app.chain import AIChain, Python doesn't know where to look for src.

Solution 1: PYTHONPATH in the workflow

      - name: Run tests
        run: pytest tests/ -v --tb=short
        env:
          PYTHONPATH: ${{ github.workspace }}

Solution 2: pyproject.toml with pythonpath

# pyproject.toml
[tool.pytest.ini_options]
testpaths = ["tests"]
pythonpath = ["."]

It works both locally and in CI — you don't need to configure anything in the workflow.

Solution 3: Install your project in editable mode

      - name: Install project
        run: pip install -e .

      - name: Run tests
        run: pytest tests/ -v --tb=short

It requires a pyproject.toml or setup.py with your package's config.

Which one to choose?

Recommendation: Use pyproject.toml with pythonpath = ["."]. It works in both environments and doesn't require changes to the workflow.

conftest.py: Shared fixtures in CI

The conftest.py file is where you define fixtures that multiple tests share. In CI, this is especially useful for configurations that depend on the environment:

# tests/conftest.py
import os
import pytest


@pytest.fixture
def api_key():
    """Return the API key, or skip if it doesn't exist."""
    key = os.environ.get("OPENAI_API_KEY")
    if not key:
        pytest.skip("OPENAI_API_KEY not set — skipping integration test")
    return key


@pytest.fixture
def sample_prompt():
    """Example prompt for tests."""
    return "Explain what machine learning is in one sentence."


@pytest.fixture
def mock_llm_response():
    """Mock response from an LLM for unit tests."""
    return {
        "content": "Machine learning is a branch of AI.",
        "model": "gpt-4o-mini",
        "tokens_used": 42,
        "cost": 0.00021,
    }

Separating unit tests from integration tests

The api_key fixture lets you separate unit tests from integration tests automatically:

# tests/test_chain.py
import pytest
from src.ai_app.chain import AIChain


def test_chain_initializes():
    chain = AIChain()
    assert chain is not None


def test_chain_validates_input(mock_llm_response):
    chain = AIChain()
    assert chain.validate("test input") is True


@pytest.mark.integration
def test_chain_calls_llm(api_key):
    """This test only runs if there's an API key."""
    chain = AIChain(api_key=api_key)
    result = chain.process("Hello")
    assert "content" in result

In CI without an API key:

tests/test_chain.py::test_chain_initializes PASSED
tests/test_chain.py::test_chain_validates_input PASSED
tests/test_chain.py::test_chain_calls_llm SKIPPED (OPENAI_API_KEY not set)

2 passed, 1 skipped in 0.15s

The unit tests always run. The integration tests get skipped when there's no API key.

Local vs CI: The differences that matter

These differences cause 90% of the "it works on my machine":

Classic error: unlisted dependency

import rich    # Installed with pip install rich, but it's not in requirements.txt

ModuleNotFoundError: No module named 'rich'

Locally it works because rich is in your virtualenv. In CI it fails because only what's in requirements.txt gets installed.

Checklist before pushing

• ✅ Are all the imports in requirements.txt?
• ✅ Are the paths relative to the project, not absolute paths from your machine?
• ✅ Do the tests that need files use fixtures or create them in the test?
• ✅ Are the required environment variables documented?
• ✅ Are there files in .gitignore that the tests need?

The complete recommended workflow

The best practice is to move the pytest configuration to pyproject.toml:

# pyproject.toml
[tool.pytest.ini_options]
testpaths = ["tests"]
pythonpath = ["."]
addopts = "-v --tb=short"

# .github/workflows/test.yml
name: Tests
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Setup Python 3.12
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: |
          # Upgrade pip first — runner's pip can be months old with known bugs
          python -m pip install --upgrade pip
          pip install -r requirements.txt

      # Bare "pytest" works because addopts in pyproject.toml injects -v --tb=short
      - name: Run tests
        run: pytest

With addopts in pyproject.toml, the -v --tb=short flags are applied automatically. The workflow stays clean and the configuration lives with the project.

Troubleshooting

"collected 0 items"

Cause: pytest didn't find any tests. The most common reasons:

1. The test files don't start with test_ (e.g. chain_tests.py instead of test_chain.py)
2. The test functions don't start with test_ (e.g. def chain_test() instead of def test_chain())
3. The tests/ path doesn't exist or is spelled differently (e.g. test/ vs tests/)

# Locally, check what pytest finds
pytest tests/ --collect-only

"ModuleNotFoundError: No module named 'src'"

Cause: PYTHONPATH isn't configured in CI.

Solution: Add pythonpath in pyproject.toml:

[tool.pytest.ini_options]
pythonpath = ["."]

Or add PYTHONPATH in the workflow:

      - name: Run tests
        run: pytest tests/ -v
        env:
          PYTHONPATH: ${{ github.workspace }}

"The test passes locally but fails in CI"

Cause: Differences between your local environment and the runner:

1. Missing dependency in requirements.txt
2. Local file that isn't in git (.gitignore)
3. Environment variable that isn't in the workflow
4. Hardcoded absolute path (/Users/your-name/...)

Solution: Reproduce the clean environment locally:

python -m venv .venv-test
source .venv-test/bin/activate
pip install -r requirements.txt
pytest tests/ -v --tb=short

If it fails in this clean venv, it fails in CI.

Exercises

Exercise 1: Your first workflow with pytest

Create a workflow that runs pytest when there's a push to main or a pull request. The project has this structure:

my-app/
├── src/
│   └── calculator.py
├── tests/
│   └── test_calculator.py
└── requirements.txt

# src/calculator.py
def add(a: float, b: float) -> float:
    return a + b

def divide(a: float, b: float) -> float:
    if b == 0:
        raise ValueError("Cannot divide by zero")
    return a / b

# tests/test_calculator.py
from src.calculator import add, divide
import pytest

def test_add_positive():
    assert add(2, 3) == 5

def test_add_negative():
    assert add(-1, -1) == -2

def test_divide_normal():
    assert divide(10, 2) == 5.0

def test_divide_by_zero():
    with pytest.raises(ValueError, match="Cannot divide by zero"):
        divide(10, 0)

# requirements.txt
pytest==8.3.4

# .github/workflows/test.yml
name: Tests
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Setup Python 3.12
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run tests
        run: pytest tests/ -v --tb=short
        env:
          PYTHONPATH: ${{ github.workspace }}

tests/test_calculator.py::test_add_positive PASSED                     [ 25%]
tests/test_calculator.py::test_add_negative PASSED                     [ 50%]
tests/test_calculator.py::test_divide_normal PASSED                    [ 75%]
tests/test_calculator.py::test_divide_by_zero PASSED                   [100%]

4 passed in 0.08s

Exercise 2: Configure pyproject.toml to remove PYTHONPATH from the workflow

Take the workflow from the previous exercise and move the pytest configuration to pyproject.toml. The workflow should have neither PYTHONPATH nor pytest flags.

# pyproject.toml
[tool.pytest.ini_options]
testpaths = ["tests"]
pythonpath = ["."]
addopts = "-v --tb=short"

# .github/workflows/test.yml
name: Tests
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Setup Python 3.12
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run tests
        run: pytest

Exercise 3: Separate unit tests from integration tests with conftest.py

You have tests that call an LLM and tests that don't. Create a conftest.py that lets you run only the unit tests in CI (without an API key) and skip the integration tests automatically.

# tests/test_ai_chain.py
import pytest

def test_prompt_format():
    """Unit test - doesn't need an API key."""
    prompt = f"Translate: {'hello'}"
    assert "Translate:" in prompt

def test_response_parsing():
    """Unit test - doesn't need an API key."""
    response = {"content": "hola", "tokens": 5}
    assert "content" in response
    assert response["tokens"] > 0

@pytest.mark.integration
def test_llm_call(api_key):
    """Integration test - needs an API key."""
    assert api_key is not None

@pytest.mark.integration
def test_full_pipeline(api_key):
    """Integration test - needs an API key."""
    assert len(api_key) > 10

# tests/conftest.py
import os
import pytest


def pytest_configure(config):
    config.addinivalue_line(
        "markers", "integration: marks tests as integration tests (require API key)"
    )


@pytest.fixture
def api_key():
    key = os.environ.get("OPENAI_API_KEY")
    if not key:
        pytest.skip("OPENAI_API_KEY not set")
    return key

tests/test_ai_chain.py::test_prompt_format PASSED                      [ 25%]
tests/test_ai_chain.py::test_response_parsing PASSED                   [ 50%]
tests/test_ai_chain.py::test_llm_call SKIPPED (OPENAI_API_KEY not set) [ 75%]
tests/test_ai_chain.py::test_full_pipeline SKIPPED (OPENAI_API_KEY...) [100%]

2 passed, 2 skipped in 0.05s

Exercise 4: Diagnose a broken workflow

This workflow fails in CI. Find the 3 errors and fix them:

# .github/workflows/test.yml
name: Tests
on: push

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Run tests
        run: pytest tests/ -v

name: Tests
on: push

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run tests
        run: pytest tests/ -v

Summary

• ✅ The 4-step pattern is always: checkout → setup python → install deps → run pytest
• ✅ Recommended flags in CI: -v --tb=short to see what failed without excessive noise
• ✅ PYTHONPATH is the most common error when moving tests to CI — solve it with pyproject.toml
• ✅ conftest.py lets you separate unit tests from integration tests automatically
• ✅ Local vs CI differ in Python version, PYTHONPATH, dependencies, environment variables, and filesystem
• ✅ The Actions logs tell you exactly what happened: look for collected, FAILED, and the error message
• ✅ pyproject.toml is better than flags in the workflow — the config lives with the project

Additional resources

1. Building and testing Python - GitHub Actions - GitHub's official guide
2. pytest Configuration - Configuring pytest with pyproject.toml
3. actions/setup-python - Complete documentation for the action
4. pytest Command Line Flags - Reference for all the flags
5. conftest.py - pytest - Shared fixtures