1. Introduction: Automated Testing in CI

Overview

In Module 1 you created your first CI workflow: a pipeline that runs on every push, does a checkout, installs dependencies, and runs a script. It works. But it has a problem: it runs your tests only once, on a single Python version, with no caching, no reports, and no protection against tests that hang. That isn't a professional testing pipeline — it's a script with extra steps.

This module turns that basic workflow into a real testing pipeline. You're going to learn to run tests on multiple Python versions simultaneously, cut CI times from minutes to seconds with caching, generate coverage reports visible on GitHub, and protect your pipeline against tests that call LLM APIs and can hang indefinitely.

Why it matters: Without reliable automated testing, every later module (AI checks, Docker, deployment) is built on a fragile base. A pipeline that doesn't cache dependencies loses minutes on every run. A pipeline without timeouts can lock up for 6 hours because of a test that calls OpenAI and never gets a response. A pipeline that only runs on Python 3.12 doesn't warn you that your code fails on 3.10 — and your production server runs 3.10. This module removes those risks.

Context: Where are we?

Position in the guide

This guide has 8 modules organized into 3 phases:

Phase 1: CI Fundamentals (Modules 1-3)
├── Module 1: Introduction to CI/CD and GitHub Actions
├── Module 2: Automated Testing in CI                  ← YOU ARE HERE
└── Module 3: AI-Specific CI Checks

Phase 2: CD & Deployment Pipelines (Modules 4-6)
├── Module 4: Secrets and Environment Management
├── Module 5: Docker in CI/CD
└── Module 6: Deployment Pipelines

Phase 3: Production Pipelines (Modules 7-8)
├── Module 7: Monitoring, Notifications and Advanced Patterns
└── Module 8: Capstone Project — Production AI Pipeline

Total estimated duration of the guide: 6-7 hours (self-paced).

Summary by module

Where are we heading?

Module 2 is the bridge between "I have a workflow that runs" and "I have a testing pipeline that protects my code". Here you build the automated testing base you'll use in all the following modules.

The progression is deliberate:

1. First you learned the fundamentals (Module 1) — without this, a workflow YAML is meaningless text
2. Now you automate real testing (this module) — the base that everything else needs
3. Then you add AI-specific checks (Module 3) — prompt regression, cost estimation
4. You protect secrets (Module 4) — secure API keys for integration tests
5. You containerize (Module 5) — Docker build and push in CI
6. You deploy (Module 6) — staging → approval → production
7. You monitor (Module 7) — notifications, scheduled runs
8. You integrate everything (Module 8) — complete commit-to-production pipeline

Module objective

By completing this module you'll be able to:

• ✅ Run pytest in GitHub Actions with dependency install and result reporting
• ✅ Implement matrix testing against Python 3.10, 3.11, and 3.12 in parallel
• ✅ Configure dependency caching that dramatically cuts CI times
• ✅ Generate test reports (JUnit XML) and coverage reports visible on GitHub
• ✅ Implement timeouts at the job level and at the individual test level for LLM tests
• ✅ Choose between fail-fast and continue-on-error depending on your need
• ✅ Distinguish unit tests (always in CI) from integration tests with an LLM (expensive and slow)

Professional objective

When your team asks you "why does CI take 8 minutes on every push?", you'll know: because it has no pip caching, it runs on a single Python version, it has no timeouts, and it reinstalls everything from scratch every time. More importantly: you'll know how to fix it in 20 minutes. That's the level of judgment this module gives you.

Prerequisites

Required knowledge

• ✅ Module 1 completed: You know how to create workflows, you understand jobs, steps, triggers
• ✅ Basic pytest: You know how to write tests with assert, fixtures, and run them locally
• ✅ requirements.txt: You know how to manage Python dependencies
• ✅ Basic Git: Push, pull, branches, PRs — GitHub Actions fires on these events
• ✅ Terminal: You move around comfortably in the terminal, you install packages with pip

Quick prerequisites check

Answer these 5 questions mentally. If you can't answer 4 out of 5, review the indicated resource before continuing.

1. What does on: push do in a GitHub Actions workflow?

2. What's the difference between a job and a step in GitHub Actions?

3. How do you run pytest locally with verbose output?

pytest tests/ -v

4. What does pip install -r requirements.txt do?

5. What is a fixture in pytest?

If you're NOT ready

Don't try to move forward without Module 1 completed. This module assumes you already have a working workflow and understand GitHub Actions' YAML structure.

Module content

Capsule 01: Module introduction (this one)

Context, objectives, prerequisites, technical setup. The base for everything that follows.

Estimated time: ~15 min

Capsule 02: pytest in GitHub Actions

Configure pytest to run automatically in CI: workflow structure, dependency install, execution with useful flags, and interpreting results in the Actions UI. You'll learn to read the runner's logs, understand what each line of the output means, and debug when a test passes locally but fails in CI.

Estimated time: ~30 min

Capsule 03: Matrix Testing (Python Versions)

Run your tests simultaneously on Python 3.10, 3.11, and 3.12. Understand why: a dependency that works on 3.11 can fail on 3.10. Matrix testing is your safety net. You'll see how GitHub Actions creates parallel jobs and how to interpret the results when one version fails and the others pass.

Estimated time: ~30 min

Capsule 04: Dependency Caching

The game-changer of CI performance. Configure actions/cache for pip and watch pip install drop from 2 minutes to 5 seconds. Understand cache keys, invalidation, and the risks of a stale cache. You'll compare runs with and without caching to see the real difference in your pipeline.

Estimated time: ~30 min

Capsule 05: Test Reports and Coverage

Generate JUnit XML reports and coverage reports that integrate with GitHub. See which tests passed, which failed, and what percentage of your code is covered — all from GitHub's UI. Configure minimum coverage thresholds so the pipeline fails if coverage drops.

Estimated time: ~45 min

Capsule 06: Timeouts for LLM Tests

The most important protection for AI pipelines. Configure timeouts at the job level and at the individual test level. A test that calls OpenAI and hangs shouldn't run for 6 hours. You'll implement the @pytest.mark.timeout decorator and the job's global timeout, and you'll understand when to use each one.

Estimated time: ~30 min

Capsule 07: Strategies for Large Test Suites

Fail-fast vs continue-on-error. Separating unit tests from integration tests. Parallelizing test suites. The decisions that matter when your test suite grows. You'll see how to organize your conftest.py and your markers so CI runs the fast stuff first and the expensive stuff after.

Estimated time: ~30 min

Capsule 08: Project — Automated Test Pipeline

Capstone project: a complete pipeline with matrix testing (3 Python versions), dependency caching, JUnit XML reports, a coverage report, and configured timeouts. You build the pipeline end to end and validate it with real pushes to GitHub.

Estimated time: ~45 min

Total module time: ~4.5 hours

This module's transformation

Mental model: before and after

BEFORE (Module 1):
┌─────────────────────────────────┐
│  push → checkout → pip install  │──→ pytest → ✅/❌
│     (2 min)    (no cache)       │   (1 version, no reports,
│                                 │    no timeout, no coverage)
└─────────────────────────────────┘
    Total: ~3-4 min | Protection: minimal


AFTER (Module 2):
┌─────────────────────────────────────────────────┐
│  push → checkout → setup-python → cache restore │
│                                    (5 sec)      │
│     ┌──── Python 3.10 ─── pytest ─── report ────┤
│     ├──── Python 3.11 ─── pytest ─── report ────┤  ← matrix
│     └──── Python 3.12 ─── pytest ─── report ────┤
│                                                  │
│     timeout: 15 min (job) + 30s (per test)      │
│     artifacts: junit.xml + coverage.xml          │
└─────────────────────────────────────────────────┘
    Total: ~1 min (with cache) | Protection: complete

The difference isn't cosmetic. It's the difference between "I hope it works" and "I know it works on 3 versions, with measured coverage, and protection against hangs".

Before (Module 1) — code

# One job, one Python version, no caching, no reports
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt    # 2 min every time
      - run: pytest tests/ -v                   # No timeout

After (Module 2) — code

# Matrix testing, caching, reports, timeouts
jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.10", "3.11", "3.12"]    # 3 versions
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
          cache: "pip"                                 # 5 seconds
      - run: pip install -r requirements.txt
      - run: pytest tests/ -v --junitxml=report.xml --cov=src  # Reports
      - uses: actions/upload-artifact@v4               # Save the reports

The difference: coverage of 3 Python versions, CI time cut by 80%, visible reports, and protection against hung tests.

Connection with the guide's project

Direct impact on later modules

Everything you learn here is used directly in the following modules:

• Module 3 (AI Checks): Adds AI-specific checks as steps to the testing pipeline you build here. Prompt regression and cost estimation run after pytest in the same workflow.
• Module 4 (Secrets): Integration tests with LLM APIs need API keys. The strategy of separating unit tests from integration tests (capsule 07) defines which tests need secrets and which don't.
• Module 5 (Docker): Docker build uses the same caching pattern you learn here. Layer caching in Actions follows the same logic as pip caching.
• Module 6 (Deployment): The deployment pipeline only runs if the testing pipeline passes. The reliability you build here determines whether you can trust the automatic deploy.
• Module 8 (Capstone): The capstone pipeline includes all of this module's testing infrastructure as its first stage.

Extensible architecture

The architecture you build here is extensible: adding a new check means adding a step, not rewriting the pipeline.

Module 2: pytest + matrix + cache + reports + timeouts
    ↓ (stays intact)
Module 3: + prompt regression step + cost estimation step
    ↓
Module 5: + docker build step + docker push step
    ↓
Module 8: complete pipeline with all the layers

Technical setup

Python and pip

Check that you have Python 3.10 or higher:

python --version
# Expected: Python 3.10.x, 3.11.x, or 3.12.x

If you have a version below 3.10, upgrade before continuing. Matrix testing requires your code to be compatible with 3.10+, and the best way to ensure that is to develop on one of the versions you're going to test.

Module dependencies

Install pytest and the plugins you'll use throughout the module:

pip install pytest>=8.0.0 pytest-cov>=5.0.0 pytest-timeout>=2.3.0

Check that each tool is installed correctly:

# Check pytest
pytest --version
# Expected: pytest 8.x.x

# Check pytest-cov
pytest --co -q --cov=. 2>&1 | head -1
# If it doesn't error, it's installed

# Check pytest-timeout
python -c "import pytest_timeout; print('pytest-timeout OK')"
# Expected: pytest-timeout OK

requirements.txt for the module

Create or update your requirements.txt with the testing dependencies:

# requirements.txt
pytest>=8.0.0
pytest-cov>=5.0.0
pytest-timeout>=2.3.0

If you already have a requirements.txt with your project's dependencies, add these three at the end. The CI workflow will install everything together.

pytest configuration with pyproject.toml

Configure pytest centrally so you don't repeat flags in every command. Create or update pyproject.toml at the root of the project:

[tool.pytest.ini_options]
testpaths = ["tests"]
addopts = "-v --tb=short"
markers = [
    "slow: tests that take more than 10 seconds",
    "integration: tests that call external APIs",
    "unit: fast isolated tests",
]
timeout = 30

This configures:

• testpaths: pytest looks for tests only in tests/ by default
• addopts: verbose output and short tracebacks automatically
• markers: custom markers to classify tests (you'll use them in capsule 07)
• timeout: each individual test has a 30-second limit (capsule 06)

Recommended file structure

your-project/
├── .github/
│   └── workflows/
│       └── ci.yml              # Main workflow (from Module 1)
├── src/
│   └── your_module/
│       ├── __init__.py
│       └── main.py
├── tests/
│   ├── __init__.py
│   ├── conftest.py             # Shared fixtures
│   ├── test_unit.py            # Unit tests (fast)
│   └── test_integration.py    # Integration tests (with LLM APIs)
├── pyproject.toml              # pytest config
├── requirements.txt            # Dependencies
└── README.md

Complete setup verification

Run this command to validate that everything is ready:

# Create a minimal test to check
mkdir -p tests
cat > tests/test_setup.py << 'EOF'
import pytest

def test_setup_works():
    assert True

@pytest.mark.timeout(5)
def test_timeout_works():
    import time
    time.sleep(1)
    assert True

def test_coverage_target():
    result = 1 + 1
    assert result == 2
EOF

# Run the tests with coverage
pytest tests/test_setup.py -v --cov=tests --timeout=10

# Expected output:
# tests/test_setup.py::test_setup_works PASSED
# tests/test_setup.py::test_timeout_works PASSED
# tests/test_setup.py::test_coverage_target PASSED
# ---------- coverage: ... ----------
# Name                     Stmts   Miss  Cover
# tests/test_setup.py          8      0   100%

If you see the 3 tests passing and the coverage report, your setup is ready for the module.

Versions and compatibility

Versions used in this guide

About Python versions in matrix testing

You're going to test against 3.10, 3.11, and 3.12. The reason: these are the three versions actively supported by CPython with security updates. Python 3.9 lost active support in October 2024. Python 3.13 is available but many AI dependencies (LangChain, OpenAI SDK) don't yet guarantee full compatibility.

If your project has a specific version requirement (e.g., your production server runs 3.11), make sure to include that version in the matrix.

About the GitHub Actions runner

ubuntu-latest is a free runner that GitHub keeps updated. It includes Python, pip, and the common tools pre-installed. The runner has 2 vCPUs, 7 GB of RAM, and 14 GB of SSD — enough for testing AI applications that use APIs (not for training models).

What this module does NOT cover

• ❌ AI-specific checks: Prompt regression testing and cost estimation are Module 3. This module builds the testing base; Module 3 adds the AI-specific checks.
• ❌ Secrets in CI: API keys for integration tests with LLM APIs are Module 4. Here you'll learn to separate unit tests from integration tests, but not to handle secrets securely.
• ❌ Docker build in CI: That's Module 5. Caching Docker layers is similar to pip caching but has its own complexities.
• ❌ Testing in depth: This guide assumes you already know pytest (from guide #13). Here we don't teach you to write tests, but to automate them in CI. We don't cover advanced fixtures, parametrize, mocking, or test design patterns.
• ❌ Alternative CI platforms: We focus exclusively on GitHub Actions. We don't cover GitLab CI, CircleCI, Jenkins, or Buildkite. The concepts (matrix, caching, timeouts) are transferable, but the syntax isn't.
• ❌ Self-hosted runners: We use GitHub's free runners. Configuring your own runners (for GPU, for example) is out of scope.

Evidence of success

By the end of this module, you'll know you succeeded if you can:

• Configure pytest to run automatically in GitHub Actions
• Run tests on 3 Python versions simultaneously with a matrix
• Cut the pip install time in CI from minutes to seconds with caching
• See test reports and coverage directly in GitHub's UI
• Configure timeouts that protect against hung tests
• Decide when to use fail-fast vs continue-on-error
• Separate unit tests from integration tests with markers

Concrete validation criteria

For each item, this is the level of evidence that demonstrates mastery:

Quick self-assessment test

If you can answer these questions by the end of the module, you're on the right track:

1. Why would you want to test on 3 Python versions if your production runs only one?
2. What happens when the cache is invalidated? How long does the first run without cache take?
3. What's the difference between timeout-minutes on the job and @pytest.mark.timeout on a test?
4. Why would you separate unit tests from integration tests in CI?
5. If your coverage drops from 80% to 65% in a PR, how would you detect it automatically?

If you check all the boxes and can answer the 5 questions → you're ready for Module 3.

Additional resources

1. pytest in GitHub Actions - Official guide
2. GitHub Actions Cache - Caching documentation
3. Matrix Strategy - Matrix reference
4. pytest-cov - Coverage plugin for pytest
5. Upload Artifact Action - Save workflow files
6. pytest-timeout - Timeout plugin for tests
7. pyproject.toml for pytest - Centralized pytest configuration