Module 2: Automated Testing in CI
8. Project — Automated Test Pipeline
Project overview
It's time to integrate everything you learned in this module into a professional testing pipeline. You're going to build a workflow that runs tests automatically on 3 Python versions, uses dependency caching to be fast, generates JUnit XML reports and coverage reports, and has timeouts configured to protect against hung tests.
The result is not an academic exercise — it's a real testing pipeline that will live in your repository and protect your code on every push. This pipeline is the foundation on top of which you'll add AI-specific checks in Module 3.
Project goal
Build an automated testing pipeline with GitHub Actions that includes matrix testing, dependency caching, test reports, coverage, and timeouts.
Prerequisites
- ✅ You completed capsules 01-07 of this module
- ✅ You have the Module 1 project with working tests
- ✅ You have access to the Actions tab of your repo on GitHub
Recap: What you built in Module 1
Before starting, make sure you're clear on what you have so far. In the Module 1 project you built:
- A repository with a
src/andtests/structure src/main.pywith the functionsgreetandclassify_sentimentthat simulate a basic AI servicesrc/utils.pywith utility functions:estimate_tokens,calculate_cost, andvalidate_prompt- Basic unit tests in
tests/test_main.pyandtests/test_utils.py - A simple
ci.ymlworkflow that ranpyteston a single Python version - A
requirements.txtwith the project's dependencies
That pipeline worked, but it was fragile. It didn't test on multiple Python versions, it didn't generate reports, it had no caching, and it didn't separate unit tests from integration tests. In this project you're going to take that foundation and turn it into a professional testing pipeline.
Technical specifications
Project structure
You're using the same project from Module 1, with these additions:
my-ai-project/
├── .github/
│ └── workflows/
│ └── ci.yml ← Updated with everything new
├── src/
│ ├── __init__.py
│ ├── main.py
│ └── utils.py
├── tests/
│ ├── __init__.py
│ ├── unit/ ← NEW: organized tests
│ │ ├── __init__.py
│ │ ├── test_main.py
│ │ └── test_utils.py
│ └── integration/ ← NEW: tests that would simulate API calls
│ ├── __init__.py
│ └── test_ai_service.py
├── requirements.txt ← Updated
├── requirements-dev.txt ← NEW: development dependencies
├── pyproject.toml ← Updated
└── README.md
Source code (from Module 1)
If for some reason you don't have these files, here is the code the tests expect. If you already have them from Module 1, verify that the function signatures match.
src/main.py
"""Main module of the AI service."""
POSITIVE_WORDS = {"great", "amazing", "excellent", "wonderful", "love", "fantastic", "awesome"}
NEGATIVE_WORDS = {"terrible", "awful", "bad", "worst", "hate", "horrible", "disappointed"}
def greet(name: str) -> str:
"""Generate a greeting for the service's user."""
cleaned = name.strip()
if not cleaned:
return "Hello, anonymous! Welcome to the AI service."
return f"Hello, {cleaned}! Welcome to the AI service."
def classify_sentiment(text: str) -> dict:
"""Classify the sentiment of a text using keyword matching.
In a real service this would call an LLM. Here we use a local
implementation so the tests don't depend on external APIs.
"""
if not text or not text.strip():
raise ValueError("Text cannot be empty")
words = set(text.lower().split())
pos_count = len(words & POSITIVE_WORDS)
neg_count = len(words & NEGATIVE_WORDS)
if pos_count > neg_count:
sentiment = "positive"
confidence = min(0.5 + pos_count * 0.15, 1.0)
elif neg_count > pos_count:
sentiment = "negative"
confidence = min(0.5 + neg_count * 0.15, 1.0)
else:
sentiment = "neutral"
confidence = 0.5
return {
"text": text,
"sentiment": sentiment,
"confidence": confidence,
}src/utils.py
"""Utility functions for the AI service."""
MODEL_PRICING = {
"gpt-4o-mini": {"input": 0.150 / 1_000_000, "output": 0.600 / 1_000_000},
"gpt-4o": {"input": 2.50 / 1_000_000, "output": 10.00 / 1_000_000},
}
def estimate_tokens(text: str) -> int:
"""Estimate the number of tokens in a text.
Uses the ~4 characters per token heuristic. It's not exact,
but it's enough for cost estimates.
"""
if not text:
return 0
return max(1, len(text) // 4)
def calculate_cost(
prompt_tokens: int, completion_tokens: int, model: str
) -> dict:
"""Calculate the estimated cost of a call to a model."""
if model not in MODEL_PRICING:
raise ValueError(f"Unknown model: {model}")
pricing = MODEL_PRICING[model]
input_cost = prompt_tokens * pricing["input"]
output_cost = completion_tokens * pricing["output"]
return {
"model": model,
"prompt_tokens": prompt_tokens,
"completion_tokens": completion_tokens,
"input_cost": round(input_cost, 8),
"output_cost": round(output_cost, 8),
"total_cost": round(input_cost + output_cost, 8),
}
def validate_prompt(prompt: str, max_tokens: int = 8000) -> dict:
"""Validate a prompt before sending it to a model."""
issues = []
if not prompt or not prompt.strip():
issues.append("Prompt is empty")
elif len(prompt.strip()) < 10:
issues.append("Prompt is too short (minimum 10 characters)")
estimated = estimate_tokens(prompt)
if estimated > max_tokens:
issues.append(f"Prompt is too long: ~{estimated} tokens (max {max_tokens})")
return {
"valid": len(issues) == 0,
"issues": issues,
"estimated_tokens": estimated,
}New and updated files
requirements-dev.txt
-r requirements.txt
pytest>=8.0
pytest-cov>=5.0
pytest-timeout>=2.3
ruff>=0.3.0
tests/unit/__init__.py
tests/unit/test_main.py
"""Unit tests for the main module."""
import pytest
from src.main import greet, classify_sentiment
class TestGreet:
def test_greet_with_name(self):
assert greet("Alice") == "Hello, Alice! Welcome to the AI service."
def test_greet_empty_string(self):
result = greet("")
assert "anonymous" in result
def test_greet_whitespace_only(self):
result = greet(" ")
assert "anonymous" in result
def test_greet_strips_whitespace(self):
result = greet(" Bob ")
assert "Bob" in result
assert " Bob " not in result
def test_greet_special_characters(self):
result = greet("María José")
assert "María José" in result
class TestClassifySentiment:
def test_positive_sentiment(self):
result = classify_sentiment("This is great and amazing!")
assert result["sentiment"] == "positive"
assert result["confidence"] > 0.5
def test_negative_sentiment(self):
result = classify_sentiment("This is terrible and awful!")
assert result["sentiment"] == "negative"
assert result["confidence"] > 0.5
def test_neutral_sentiment(self):
result = classify_sentiment("The sky is blue.")
assert result["sentiment"] == "neutral"
assert result["confidence"] == 0.5
def test_empty_text_raises(self):
with pytest.raises(ValueError, match="Text cannot be empty"):
classify_sentiment("")
def test_result_structure(self):
result = classify_sentiment("Hello world")
assert "text" in result
assert "sentiment" in result
assert "confidence" in result
def test_confidence_range(self):
result = classify_sentiment("great amazing excellent love")
assert 0 <= result["confidence"] <= 1.0
def test_result_includes_original_text(self):
text = "Test input text"
result = classify_sentiment(text)
assert result["text"] == texttests/unit/test_utils.py
"""Unit tests for the utility functions."""
import pytest
from src.utils import estimate_tokens, calculate_cost, validate_prompt
class TestEstimateTokens:
def test_empty_string(self):
assert estimate_tokens("") == 0
def test_short_text(self):
result = estimate_tokens("Hello")
assert result >= 1
def test_longer_text(self):
text = "This is a longer piece of text for token estimation."
result = estimate_tokens(text)
assert 10 <= result <= 20
def test_returns_integer(self):
assert isinstance(estimate_tokens("test"), int)
def test_minimum_one_token(self):
assert estimate_tokens("Hi") >= 1
class TestCalculateCost:
def test_gpt4o_mini_cost(self):
result = calculate_cost(1000, 500, "gpt-4o-mini")
assert result["model"] == "gpt-4o-mini"
assert result["total_cost"] > 0
def test_gpt4o_cost(self):
result = calculate_cost(1000, 500, "gpt-4o")
assert result["total_cost"] > calculate_cost(1000, 500, "gpt-4o-mini")["total_cost"]
def test_unknown_model_raises(self):
with pytest.raises(ValueError, match="Unknown model"):
calculate_cost(100, 100, "gpt-5-imaginary")
def test_zero_tokens(self):
result = calculate_cost(0, 0, "gpt-4o-mini")
assert result["total_cost"] == 0
def test_cost_structure(self):
result = calculate_cost(1000, 500, "gpt-4o")
required_keys = [
"model", "prompt_tokens", "completion_tokens",
"input_cost", "output_cost", "total_cost",
]
for key in required_keys:
assert key in result
def test_cost_proportional_to_tokens(self):
cost_1k = calculate_cost(1000, 500, "gpt-4o-mini")["total_cost"]
cost_2k = calculate_cost(2000, 1000, "gpt-4o-mini")["total_cost"]
assert abs(cost_2k - cost_1k * 2) < 0.0001
class TestValidatePrompt:
def test_valid_prompt(self):
result = validate_prompt("Summarize the following text in 3 sentences.")
assert result["valid"] is True
assert len(result["issues"]) == 0
def test_empty_prompt(self):
result = validate_prompt("")
assert result["valid"] is False
def test_short_prompt(self):
result = validate_prompt("Hi")
assert result["valid"] is False
def test_token_estimation_included(self):
result = validate_prompt("A normal prompt for testing purposes.")
assert "estimated_tokens" in result
assert result["estimated_tokens"] > 0
def test_very_long_prompt(self):
long_prompt = "word " * 20000
result = validate_prompt(long_prompt, max_tokens=4000)
assert result["valid"] is False
assert any("too long" in issue.lower() for issue in result["issues"])tests/integration/__init__.py
tests/integration/test_ai_service.py
"""
Integration tests that simulate interactions with AI services.
In a real project, these tests would call external APIs.
Here we simulate the pattern to demonstrate timeouts and separation.
"""
import time
import pytest
from src.main import classify_sentiment
from src.utils import estimate_tokens, calculate_cost
@pytest.mark.integration
@pytest.mark.timeout(10)
class TestAIServiceIntegration:
"""Tests that simulate calls to AI services."""
def test_sentiment_pipeline(self):
"""Simulates the full pipeline: tokenize → classify → cost."""
text = "This product is absolutely amazing and wonderful!"
tokens = estimate_tokens(text)
sentiment = classify_sentiment(text)
cost = calculate_cost(tokens, tokens // 2, "gpt-4o-mini")
assert sentiment["sentiment"] == "positive"
assert cost["total_cost"] > 0
assert cost["total_cost"] < 0.01
def test_batch_processing(self):
"""Simulates batch processing of multiple texts."""
texts = [
"Great product, love it!",
"Terrible experience, worst ever.",
"The weather is nice today.",
"Amazing service, highly recommend!",
"Bad quality, very disappointed.",
]
results = [classify_sentiment(text) for text in texts]
positive = sum(1 for r in results if r["sentiment"] == "positive")
negative = sum(1 for r in results if r["sentiment"] == "negative")
assert positive >= 2
assert negative >= 2
def test_cost_estimation_batch(self):
"""Simulates cost estimation for a batch of requests."""
texts = ["Sample text for estimation."] * 100
total_cost = 0
for text in texts:
tokens = estimate_tokens(text)
cost = calculate_cost(tokens, tokens, "gpt-4o-mini")
total_cost += cost["total_cost"]
assert total_cost > 0
assert total_cost < 1.0
@pytest.mark.timeout(5)
def test_with_simulated_latency(self):
"""Simulates a test with API latency (timeout test)."""
time.sleep(0.5)
result = classify_sentiment("Quick test with simulated latency")
assert result["sentiment"] in ["positive", "negative", "neutral"]pyproject.toml (updated)
[tool.ruff]
target-version = "py310"
line-length = 88
[tool.ruff.lint]
select = ["E", "F", "I", "W"]
[tool.pytest.ini_options]
testpaths = ["tests"]
pythonpath = ["."]
markers = [
"integration: tests that simulate external API calls",
"slow: tests that take more than 5 seconds",
]
timeout = 30
[tool.coverage.run]
source = ["src"]
omit = ["tests/*"]
[tool.coverage.report]
show_missing = true
fail_under = 80The Workflow: the complete ci.yml
This is the workflow that integrates everything you learned in the module:
name: CI Pipeline
on:
push:
branches: [main]
paths-ignore:
- "*.md"
- "docs/**"
pull_request:
branches: [main]
workflow_dispatch:
env:
MIN_COVERAGE: 80
jobs:
# ──────────── JOB 1: LINT ────────────
lint:
name: Code Quality
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.12"
cache: "pip"
- name: Install linting tools
run: pip install ruff
- name: Run linting
run: ruff check src/ tests/
- name: Check formatting
run: ruff format --check src/ tests/
# ──────────── JOB 2: UNIT TESTS (MATRIX) ────────────
unit-tests:
name: Unit Tests (Python ${{ matrix.python-version }})
runs-on: ubuntu-latest
timeout-minutes: 10
strategy:
fail-fast: false
matrix:
python-version: ["3.10", "3.11", "3.12"]
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
cache: "pip"
- name: Install dependencies
run: pip install -r requirements-dev.txt
- name: Run unit tests with coverage
run: |
pytest tests/unit/ \
-v \
--tb=short \
--timeout=30 \
--junitxml=reports/junit-${{ matrix.python-version }}.xml \
--cov=src \
--cov-report=term-missing \
--cov-report=xml:reports/coverage-${{ matrix.python-version }}.xml
- name: Upload test reports
if: always()
uses: actions/upload-artifact@v4
with:
name: test-reports-py${{ matrix.python-version }}
path: reports/
retention-days: 7
# ──────────── JOB 3: INTEGRATION TESTS ────────────
integration-tests:
name: Integration Tests
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.12"
cache: "pip"
- name: Install dependencies
run: pip install -r requirements-dev.txt
- name: Run integration tests
run: |
pytest tests/integration/ \
-v \
--tb=short \
--timeout=60 \
-m integration \
--junitxml=reports/junit-integration.xml
- name: Upload integration reports
if: always()
uses: actions/upload-artifact@v4
with:
name: integration-reports
path: reports/
retention-days: 7
# ──────────── JOB 4: PIPELINE STATUS ────────────
pipeline-status:
name: Pipeline Status
needs: [lint, unit-tests, integration-tests]
runs-on: ubuntu-latest
timeout-minutes: 5
if: always()
steps:
- name: Check all jobs
run: |
echo "=== Pipeline Results ==="
echo "Lint: ${{ needs.lint.result }}"
echo "Unit Tests: ${{ needs.unit-tests.result }}"
echo "Integration Tests: ${{ needs.integration-tests.result }}"
echo ""
if [ "${{ needs.lint.result }}" != "success" ] || \
[ "${{ needs.unit-tests.result }}" != "success" ] || \
[ "${{ needs.integration-tests.result }}" != "success" ]; then
echo "❌ Pipeline FAILED"
exit 1
fi
echo "✅ All checks passed!"Pipeline architecture
Push/PR
│
├──► lint ────────────────────────────────────────┐
│ │
├──► unit-tests (Python 3.10) ────────────────────┤
├──► unit-tests (Python 3.11) ────────────────────┤ (all in parallel)
├──► unit-tests (Python 3.12) ────────────────────┤
│ │
├──► integration-tests ───────────────────────────┤
│ │
└──► pipeline-status ◄────────────────────────────┘ (waits for all)
needs: [lint, unit-tests, integration-tests]
6 jobs in parallel: lint + 3 matrix (unit tests) + integration tests, followed by a final status check.
Step by step: Implementation
Step 1: Update the project structure
# Create the organized test directories
mkdir -p tests/unit tests/integration
# Move the existing tests
mv tests/test_main.py tests/unit/test_main.py
mv tests/test_utils.py tests/unit/test_utils.py
# Create the __init__.py files
touch tests/unit/__init__.py
touch tests/integration/__init__.pyStep 2: Create the new files
Create requirements-dev.txt, update pyproject.toml, and create tests/integration/test_ai_service.py with the content shown above.
Step 3: Verify locally
# Install the development dependencies
pip install -r requirements-dev.txt
# Run the unit tests
pytest tests/unit/ -v --cov=src --cov-report=term-missingExpected output:
tests/unit/test_main.py::TestGreet::test_greet_with_name PASSED
tests/unit/test_main.py::TestGreet::test_greet_empty_string PASSED
... (all the tests)
tests/unit/test_utils.py::TestValidatePrompt::test_very_long_prompt PASSED
---------- coverage: platform linux, python 3.12.0 ----------
Name Stmts Miss Cover Missing
------------------------------------------------
src/__init__.py 0 0 100%
src/main.py 25 2 92% 42-43
src/utils.py 30 0 100%
------------------------------------------------
TOTAL 55 2 96%
========================= 27 passed in 0.15s =========================
# Run the integration tests
pytest tests/integration/ -v -m integration --timeout=30Expected output:
tests/integration/test_ai_service.py::TestAIServiceIntegration::test_sentiment_pipeline PASSED
tests/integration/test_ai_service.py::TestAIServiceIntegration::test_batch_processing PASSED
tests/integration/test_ai_service.py::TestAIServiceIntegration::test_cost_estimation_batch PASSED
tests/integration/test_ai_service.py::TestAIServiceIntegration::test_with_simulated_latency PASSED
========================= 4 passed in 0.62s =========================
# Run the linter
ruff check src/ tests/
ruff format --check src/ tests/Step 4: Update the workflow and push
Replace .github/workflows/ci.yml with the complete workflow shown above.
git add .
git commit -m "Upgrade CI with matrix testing, caching, coverage, and timeouts"
git push origin mainStep 5: Verify on GitHub
- Go to Actions → you'll see 6 jobs running
- lint, 3 unit-tests (matrix), and integration-tests run in parallel
- pipeline-status waits for all of them
- Download the artifacts: reports with JUnit XML and coverage
Completeness checklist
Workflow
- Matrix testing with Python 3.10, 3.11, 3.12
-
fail-fast: falseon the matrix - Dependency caching with
cache: "pip" - Timeouts on every job
- JUnit XML reports generated
- Coverage reports generated
- Artifacts uploaded with
actions/upload-artifact - Unit tests and integration tests in separate jobs
- A final job that verifies all the results
Tests
- Unit tests organized in
tests/unit/ - Integration tests in
tests/integration/with@pytest.mark.integration - Timeouts configured:
@pytest.mark.timeout()on the integration tests - Coverage > 80% on the unit tests
- All the tests pass locally on Python 3.10, 3.11, and 3.12
Configuration
-
pyproject.tomlwith markers, a default timeout, and coverage config -
requirements-dev.txtwith pytest, pytest-cov, pytest-timeout, ruff -
.github/workflows/ci.ymlupdated with the complete pipeline
Project troubleshooting
"The matrix job fails on Python 3.10 but passes on 3.12"
Check whether your code uses Python 3.11+ features (like match statements or ExceptionGroup). The target version in pyproject.toml must be py310 if you want compatibility.
"The coverage report shows < 80%"
If --cov-fail-under=80 fails, check which lines aren't covered with --cov-report=term-missing. Add tests for the missing lines or adjust the threshold.
"The artifact upload fails"
Verify that the reports/ directory exists. Add it before the pytest step:
- name: Create reports directory
run: mkdir -p reports"The integration tests take a long time"
test_with_simulated_latency has a time.sleep(0.5). In a real project with API calls, the times can be higher. Verify that the job's timeout-minutes: 15 is enough.
"Import path issues: ModuleNotFoundError: No module named 'src'"
This is one of the most common errors. pytest needs to know where to look for the src module. The solution lives in pyproject.toml:
[tool.pytest.ini_options]
pythonpath = ["."]Without that line, pytest doesn't find src.main or src.utils. If you already have it and it still fails, verify that you're running pytest from the project root (where pyproject.toml lives), not from inside tests/ or another subdirectory.
In the GitHub Actions workflow this usually isn't a problem because actions/checkout leaves you at the repo root. But if you run it locally, make sure you're in the right directory.
"The Ruff formatting check fails in CI but passes locally"
The lint job runs ruff format --check, which verifies but doesn't modify files. If it fails, it means there are unformatted files. Before pushing:
ruff format src/ tests/
ruff check src/ tests/ --fixThe most common cause is editing files without having format-on-save configured in your editor. If you use VS Code, make sure you have the Ruff extension installed and editor.formatOnSave enabled. Another frequent cause: formatting with black instead of ruff — the two produce slightly different output.
Success criteria
| Criterion | Status |
|---|---|
| 6 jobs run in the Actions UI | ☐ |
| Matrix: 3 Python versions in parallel | ☐ |
| Caching: pip install < 10 seconds (after the first run) | ☐ |
| Reports: JUnit XML and coverage as downloadable artifacts | ☐ |
| Timeouts: configured on jobs and tests | ☐ |
| Coverage: > 80% on the unit tests | ☐ |
| Pipeline status: a final job verifies everything | ☐ |
What you built
By completing this project you have:
- ✅ A professional testing pipeline with matrix testing (3 Python versions)
- ✅ Dependency caching that makes your CI fast
- ✅ Test reports and coverage visible as artifacts
- ✅ Timeouts that protect against hung tests
- ✅ A clear separation between unit tests and integration tests
- ✅ The foundation for adding AI-specific checks in Module 3
Optional extensions
If you finished the project and want to go further, here are three extensions that reinforce what you learned and make your pipeline more robust.
1. Add mypy for type checking
Add a static type checking step to the lint job. This verifies that the type hints in your code are consistent:
- name: Install type checker
run: pip install mypy
- name: Run type checking
run: mypy src/ --ignore-missing-importsYou'll need to add mypy>=1.8 to requirements-dev.txt. mypy can be noisy at first — use --ignore-missing-imports to avoid errors from external dependencies without stubs.
2. Add pre-commit hooks
pre-commit lets you run checks before every local commit, catching errors before they reach CI. Create .pre-commit-config.yaml at the root:
repos:
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.3.0
hooks:
- id: ruff
args: [--fix]
- id: ruff-format
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.5.0
hooks:
- id: trailing-whitespace
- id: end-of-file-fixerInstall it with pip install pre-commit && pre-commit install. From now on, every git commit will run these checks automatically.
3. A CI badge in the README
Add a badge at the top of your README.md that shows the pipeline's status:
Replace YOUR-USERNAME and YOUR-REPO with your own details. The badge updates automatically — green if the pipeline passes, red if it fails. It's an immediate visual indicator of your project's health.
Next module
Module 3 (AI-Specific CI Checks) adds what makes this guide special:
- Prompt regression testing: Detecting that a change degraded the quality of the responses
- Cost estimation checks: Calculating how much a prompt will cost before merging
- Quality gates: Blocking the merge if the AI checks fail
The transition is direct: "Your tests already run automatically with matrix, caching, and reports → now let's add checks that no generic CI has."
Additional resources
- Building and testing Python - Official guide to Python in Actions
- Using a matrix for your jobs - Matrix strategy reference
- Caching dependencies - Caching guide
- pytest-cov documentation - Coverage plugin
- pytest-timeout documentation - Timeout plugin
- Upload artifact action - Saving files from workflow runs