Module 2: Automated Testing in CI

5. Test Reports and Coverage

Overview

Your tests already run automatically in CI. The workflow executes pytest, you see a green or red check on GitHub, and if it fails, you check the logs. It works, but it has a problem: all the information is buried in plain-text logs. If you want to know which tests failed, you have to read lines of output. If you want to know what percentage of your code is covered by tests, you have no idea.

Test reports and coverage reports solve this. You generate structured files (JUnit XML for results, HTML/XML for coverage) that GitHub can interpret, visualize, and store as artifacts. The result: you see which tests passed directly in a PR, you download detailed coverage reports, and you configure minimum thresholds that block the merge if coverage drops.

In the context of AI applications, coverage has an important nuance: the functions that call the LLM typically have low coverage if you use mocks. That's fine — the real integration with the LLM is validated in integration tests, not in unit tests. This capsule teaches you to generate, interpret, and act on these reports professionally.

JUnit XML: The standard test report format

JUnit XML is a standard format for reporting test results. Originally created for Java, it became the de facto standard that every CI tool understands. pytest generates it natively with a single flag.

pytest tests/ -v --junitxml=report.xml

The report.xml file contains structured information about every test:

<?xml version="1.0" encoding="utf-8"?>
<testsuites>
  <testsuite name="pytest" errors="0" failures="1" skipped="0" tests="8" time="0.45">
    <testcase classname="tests.test_main" name="test_greet" time="0.001"/>
    <testcase classname="tests.test_api" name="test_format_prompt" time="0.002"/>
    <testcase classname="tests.test_api" name="test_parse_completion_error" time="0.012">
      <failure message="AssertionError: assert 'error' not in response">
        def test_parse_completion_error():
            response = {"error": "invalid_request"}
        >   assert "error" not in response
        E   AssertionError: assert 'error' not in response
      </failure>
    </testcase>
  </testsuite>
</testsuites>

Each <testcase> includes: name, execution time, and if it failed, the details of the failure. This structure lets CI tools display results in a rich way.

Coverage reports with pytest-cov

Initial setup

pytest-cov is the standard plugin for measuring coverage in pytest, based on coverage.py.

pip install pytest-cov

Your requirements.txt should include:

pytest>=8.0
pytest-cov>=5.0

Generating coverage in the terminal

# Basic coverage
pytest tests/ --cov=src

# With detail of the missing lines
pytest tests/ --cov=src --cov-report=term-missing

# Combining it with JUnit XML
pytest tests/ -v --junitxml=report.xml --cov=src --cov-report=term-missing

Output with --cov-report=term-missing:

---------- coverage: platform linux, python 3.12.0 -----------
Name                    Stmts   Miss  Cover   Missing
-----------------------------------------------------
src/__init__.py             0      0   100%
src/main.py                12      0   100%
src/api.py                 45      8    82%   34-38, 67-70
src/prompts.py             23      5    78%   45-49
src/config.py               8      0   100%
-----------------------------------------------------
TOTAL                      88     13    85%

Column	Meaning
Stmts	Executable lines of code (statements)
Miss	Lines that no test executed
Cover	Coverage percentage (Stmts - Miss) / Stmts
Missing	Line numbers that aren't covered

The Missing column is the most actionable one: it tells you exactly which lines need tests.

Multiple formats at once

You can generate several report formats in a single command:

pytest tests/ \
  --cov=src \
  --cov-report=term-missing \
  --cov-report=html:coverage-html \
  --cov-report=xml:coverage.xml \
  --junitxml=report.xml

This generates:

report.xml — JUnit XML with the test results
coverage.xml — Coverage in XML format (for CI tools)
coverage-html/ — Directory with a browsable HTML report (with lines colored green/red)
Terminal output with the missing lines

Configuration in pyproject.toml

Instead of passing long flags every time, configure pytest-cov in pyproject.toml:

[tool.pytest.ini_options]
testpaths = ["tests"]
addopts = "-v --tb=short"

[tool.coverage.run]
source = ["src"]
omit = [
    "src/__pycache__/*",
    "src/scripts/*",
]

[tool.coverage.report]
show_missing = true
fail_under = 80
exclude_lines = [
    "pragma: no cover",
    "if __name__ == .__main__.",
    "if TYPE_CHECKING:",
    "raise NotImplementedError",
]

[tool.coverage.html]
directory = "coverage-html"

With this configuration, the command gets simpler:

# Before (without pyproject.toml)
pytest tests/ --cov=src --cov-report=term-missing --cov-report=html:coverage-html --cov-fail-under=80

# After (with pyproject.toml)
pytest --cov --cov-report=term-missing --cov-report=html

Coverage thresholds: `--cov-fail-under`

A threshold defines the minimum coverage percentage your project must maintain. If coverage drops below that threshold, pytest fails with exit code 2, causing the CI pipeline to fail.

# On the command line
pytest tests/ --cov=src --cov-fail-under=80

# Or in pyproject.toml (recommended)
# [tool.coverage.report]
# fail_under = 80

When coverage is below the threshold:

TOTAL                      88     30    66%

FAIL Required test coverage of 80% not reached. Total coverage: 65.91%

The pipeline fails even though all the tests passed — coverage doesn't reach the threshold.

Choosing the right threshold

Project type	Recommended threshold	Reason
Pure utility/library	90%	Deterministic code, easy to test
Standard REST API	80-85%	Some error paths are hard to cover
App with AI integration	70-80%	LLM calls get mocked, reducing real coverage
Prototype/MVP	60-70%	A balance between speed and quality

Coverage in the AI context: The important nuance

Consider this function and its test with a mock:

# src/ai_service.py
import openai

def generate_summary(text: str, max_tokens: int = 200) -> str:
    if not text.strip():
        raise ValueError("Text cannot be empty")

    if len(text) < 50:
        return text

    client = openai.OpenAI()
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Summarize the following text concisely."},
            {"role": "user", "content": text},
        ],
        max_tokens=max_tokens,
        temperature=0.3,
    )

    summary = response.choices[0].message.content
    if not summary:
        raise RuntimeError("LLM returned empty response")

    return summary.strip()

# tests/test_ai_service.py
from unittest.mock import patch, MagicMock
from src.ai_service import generate_summary

def test_generate_summary_empty_text():
    import pytest
    with pytest.raises(ValueError, match="Text cannot be empty"):
        generate_summary("")

@patch("src.ai_service.openai.OpenAI")
def test_generate_summary_with_mock(mock_openai_class):
    mock_client = MagicMock()
    mock_openai_class.return_value = mock_client
    mock_client.chat.completions.create.return_value = MagicMock(
        choices=[MagicMock(message=MagicMock(content="This is a summary."))]
    )

    result = generate_summary("A" * 100)
    assert result == "This is a summary."
    mock_client.chat.completions.create.assert_called_once()

Coverage will say 100%, but you are not testing the real interaction with OpenAI. The mock replaces the real call. The rule:

Unit tests with mocks → They validate your logic (parsing, validation, control flow)
Integration tests with the real API → They validate the integration with the LLM (expensive, slow)
Coverage report → It reflects the unit tests, not the integration tests

Don't inflate your threshold to compensate. A threshold of 70-80% for an AI app is reasonable and honest.

Complete workflow: Reports and coverage in GitHub Actions

# .github/workflows/test-reports.yml
name: Tests with Reports

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    # Prevents runaway LLM or install steps from burning your free-tier minutes
    timeout-minutes: 15
    # Minimum permissions for dorny/test-reporter to annotate PRs with test results
    permissions:
      contents: read
      checks: write
      pull-requests: write

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"

      - run: pip install -r requirements.txt

      # cov-fail-under makes the pipeline fail if coverage drops — catches "added feature without tests"
      - name: Run tests with coverage
        run: |
          pytest tests/ \
            -v \
            --junitxml=test-results/report.xml \
            --cov=src \
            --cov-report=term-missing \
            --cov-report=html:coverage-html \
            --cov-report=xml:coverage-html/coverage.xml \
            --cov-fail-under=80

      # if: always() is key — without it, failed tests prevent report upload (exactly when you need it most)
      - name: Publish test results
        if: always()
        uses: dorny/test-reporter@v1
        with:
          name: pytest results
          path: test-results/report.xml
          reporter: java-junit

      - name: Upload test results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: test-results
          path: test-results/
          retention-days: 30

      - name: Upload coverage report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: coverage-report
          path: coverage-html/
          retention-days: 30

Key details:

if: always() on the upload steps: Without this, when the tests fail, the artifacts don't get uploaded. With always(), they always get uploaded — which is exactly when you need the reports most.
retention-days: 30: A good balance between keeping history and not accumulating storage.
dorny/test-reporter: Publishes results directly in the PR's "Checks" tab, without needing to open the logs.

Native alternative: Job Summary (without third-party actions)

- name: Generate coverage summary
  if: always()
  run: |
    echo "## Test Coverage Report" >> $GITHUB_STEP_SUMMARY
    echo "" >> $GITHUB_STEP_SUMMARY
    echo '```' >> $GITHUB_STEP_SUMMARY
    pytest tests/ --cov=src --cov-report=term-missing 2>&1 | tail -20 >> $GITHUB_STEP_SUMMARY
    echo '```' >> $GITHUB_STEP_SUMMARY

$GITHUB_STEP_SUMMARY is a special file that GitHub Actions renders as Markdown at the end of the workflow run.

Advanced workflow: Coverage split by test type

In large projects, it's useful to separate the tests from the report generation:

name: CI with Coverage

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  unit-tests:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
      - run: pip install -r requirements.txt
      - name: Run unit tests with coverage
        run: |
          pytest tests/unit/ \
            -v \
            --junitxml=test-results/unit-report.xml \
            --cov=src \
            --cov-report=term-missing \
            --cov-report=xml:coverage.xml \
            --cov-fail-under=80
      - name: Upload unit test results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: unit-test-results
          path: |
            test-results/
            coverage.xml
          retention-days: 30

  integration-tests:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    if: github.event_name == 'pull_request'
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
      - run: pip install -r requirements.txt
      - name: Run integration tests
        run: |
          pytest tests/integration/ \
            -v \
            --junitxml=test-results/integration-report.xml \
            --timeout=60
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      - name: Upload integration test results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: integration-test-results
          path: test-results/
          retention-days: 30

The separation lets you:

Run unit tests on every push with a coverage threshold
Run integration tests only on PRs (because they need API keys and are slower)
Diagnose more easily thanks to separate artifacts

Artifacts: Saving and downloading reports

Upload artifact takes files from the runner and stores them on GitHub, associated with the workflow run.

- name: Upload coverage HTML
  uses: actions/upload-artifact@v4
  with:
    name: coverage-report
    path: coverage-html/
    retention-days: 30
    if-no-files-found: warn

Downloading with the GitHub CLI:

# List the artifacts of the latest run
gh run view --json artifacts

# Download a specific artifact
gh run download <run-id> -n coverage-report

Useful artifacts for AI projects:

Artifact	Contents	When to use it
`test-results`	JUnit XML	Always — diagnosing failures
`coverage-report`	HTML coverage	Always — coverage analysis
`llm-responses`	Logs of the LLM's responses	Debugging integration tests
`evaluation-metrics`	JSON with AI metrics	Tracking model quality

Comparisons

`--cov-report` formats

Format	Contents	When to use it
`term`	Table with percentages per file	Quick review in the logs
`term-missing`	Table + missing lines	Debugging in CI
`html`	Browsable, with colors per line	Detailed local analysis
`xml`	Coverage in Cobertura XML format	Integration with CI tools

JUnit XML vs terminal output

Aspect	Terminal output	JUnit XML
Human-readable	✅ Yes	❌ Not directly
Machine-readable	❌ No	✅ Yes
Integrates with PR checks	❌ No	✅ Yes (with actions)
Persistence	❌ Only in the logs	✅ As an artifact

Recommendation for fail_under: Define it in pyproject.toml, not as a flag. A single place for the configuration.

Troubleshooting

1. "No data was collected" — Empty coverage report

Symptom: Coverage shows TOTAL 0 0 100%.

Cause: --cov=src points to a directory that doesn't exist or doesn't contain the imported files.

Solution:

ls src/
pytest tests/ --cov=app  # or --cov=mypackage or --cov=.

2. Coverage differs between local and CI

Symptom: Local shows 90%, CI shows 75%.

Cause: Different files in src/. CI may include files you don't have locally.

Solution:

- name: Debug coverage sources
  run: |
    echo "Files in src/:"
    find src/ -name "*.py" | sort

3. Upload artifact fails with "No files were found"

Cause: The directory wasn't created because pytest failed before generating the report.

Solution:

- name: Create results directory
  run: mkdir -p test-results

- name: Run tests
  continue-on-error: true
  run: pytest tests/ --junitxml=test-results/report.xml

- name: Upload results
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: test-results
    path: test-results/
    if-no-files-found: warn

Exercises

Exercise 1: Add coverage to an existing workflow

You have this basic workflow. Modify it to add JUnit XML reports and coverage with an 80% threshold:

name: CI
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: pytest tests/ -v

See solution

name: CI
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
      - run: pip install -r requirements.txt
      - name: Run tests with coverage
        run: |
          pytest tests/ \
            -v \
            --junitxml=test-results/report.xml \
            --cov=src \
            --cov-report=term-missing \
            --cov-report=html:coverage-html \
            --cov-fail-under=80
      - name: Upload test results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: test-results
          path: test-results/
          retention-days: 30
      - name: Upload coverage report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: coverage-html
          path: coverage-html/
          retention-days: 30

Key changes:

--junitxml for structured test results
--cov=src + --cov-report=term-missing for detailed coverage
--cov-fail-under=80 to block if it drops below 80%
if: always() on the uploads so they get uploaded even when the tests fail

Exercise 2: Configure pyproject.toml for coverage

Write the pyproject.toml section that configures coverage for an AI project with this structure:

my-ai-app/
├── src/
│   ├── main.py
│   ├── llm_service.py
│   └── scripts/
│       └── seed_data.py
├── tests/
└── pyproject.toml

Requirements: measure only src/, exclude scripts/, a 75% threshold, exclude lines with pragma: no cover.

See solution

[tool.pytest.ini_options]
testpaths = ["tests"]
addopts = "-v --tb=short"
markers = [
    "slow: marks tests as slow",
    "integration: marks tests that need API keys",
    "llm: marks tests that call LLM APIs",
]

[tool.coverage.run]
source = ["src"]
omit = [
    "src/scripts/*",
    "src/__pycache__/*",
]

[tool.coverage.report]
show_missing = true
fail_under = 75
exclude_lines = [
    "pragma: no cover",
    "if __name__ == .__main__.",
    "if TYPE_CHECKING:",
    "raise NotImplementedError",
]

[tool.coverage.html]
directory = "coverage-html"
show_contexts = true

fail_under = 75 reflects that an AI app with mocks won't have coverage as high as a pure library. show_contexts = true in the HTML shows which test covered each line.

Exercise 3: Interpret a coverage report

Analyze this output and answer: (a) Which file needs the most attention? (b) Would it pass an 80% threshold? (c) What would you do to improve coverage?

---------- coverage: platform linux, python 3.12.0 -----------
Name                      Stmts   Miss  Cover   Missing
--------------------------------------------------------
src/__init__.py               0      0   100%
src/main.py                  15      0   100%
src/api.py                   42      3    93%   78-80
src/llm_service.py           35     12    66%   20-25, 40-48, 55
src/prompts.py               28      2    93%   45-46
src/config.py                10      0   100%
--------------------------------------------------------
TOTAL                       130     17    87%

See solution

(a) src/llm_service.py, with 66% coverage. It has 12 uncovered lines out of 35 total. The ranges 20-25 and 40-48 suggest entire untested blocks (probably LLM error handling or retry logic).

(b) Yes. The total is 87%, above 80%.

(c) Actions:

Review lines 20-25 of llm_service.py — probably input validation, testable without a mock
Review lines 40-48 — probably error handling (API timeout, rate limit), testable by mocking exceptions
For api.py (78-80) and prompts.py (45-46), with so few misses they're easy to cover with 1-2 tests

from unittest.mock import patch, MagicMock
import openai
import pytest

@patch("src.llm_service.openai.OpenAI")
def test_llm_service_api_timeout(mock_openai_class):
    mock_client = MagicMock()
    mock_openai_class.return_value = mock_client
    mock_client.chat.completions.create.side_effect = openai.APITimeoutError("timeout")

    with pytest.raises(openai.APITimeoutError):
        generate_summary("A" * 100)

Exercise 4: Coverage + Job Summary

Write a GitHub Actions step that: (1) runs the tests with coverage, (2) saves the percentage in a variable, and (3) publishes a summary in the Job Summary.

See solution

- name: Run tests with coverage
  id: coverage
  run: |
    pytest tests/ \
      --cov=src \
      --cov-report=term-missing \
      --junitxml=test-results/report.xml \
      2>&1 | tee pytest-output.txt

    COVERAGE_PCT=$(grep "^TOTAL" pytest-output.txt | awk '{print $4}' | tr -d '%')
    echo "coverage_pct=$COVERAGE_PCT" >> $GITHUB_OUTPUT

- name: Publish coverage summary
  if: always()
  run: |
    COVERAGE=${{ steps.coverage.outputs.coverage_pct }}

    echo "## Test Coverage Report" >> $GITHUB_STEP_SUMMARY
    echo "" >> $GITHUB_STEP_SUMMARY

    if [ "$COVERAGE" -ge 80 ]; then
      echo "**Coverage: ${COVERAGE}%** (threshold: 80%)" >> $GITHUB_STEP_SUMMARY
    else
      echo "**Coverage: ${COVERAGE}%** — below threshold of 80%" >> $GITHUB_STEP_SUMMARY
    fi

    echo "" >> $GITHUB_STEP_SUMMARY
    echo '```' >> $GITHUB_STEP_SUMMARY
    grep -A 100 "^Name" pytest-output.txt | grep -B 100 "^TOTAL" >> $GITHUB_STEP_SUMMARY
    echo '```' >> $GITHUB_STEP_SUMMARY

tee pytest-output.txt saves the output so you can process it afterward
$GITHUB_OUTPUT passes values between steps
$GITHUB_STEP_SUMMARY generates Markdown visible in the run's UI

Summary

✅ JUnit XML (--junitxml=report.xml) generates test reports that CI tools can interpret
✅ pytest-cov (--cov=src --cov-report=term-missing) measures what percentage of your code the tests execute
✅ --cov-fail-under=80 blocks the pipeline if coverage drops below the threshold
✅ pyproject.toml centralizes the coverage configuration: source, omit, threshold, exclusions
✅ actions/upload-artifact stores reports as artifacts downloadable from GitHub
✅ if: always() on uploads guarantees the reports get uploaded even when the tests fail
✅ Coverage in AI apps will be low for functions with LLM mocks — a 70-80% threshold is honest
✅ Multiple formats can be generated in a single command: term, html, xml

Additional resources

pytest-cov documentation — Complete reference for the coverage plugin
Coverage.py configuration — All the configuration options in pyproject.toml
actions/upload-artifact — Official documentation of the action for saving artifacts
dorny/test-reporter — Action for publishing test results in PR checks
GitHub Actions Job Summaries — How to write Markdown in $GITHUB_STEP_SUMMARY

← Previous4. Dependency Caching Next →6. Timeouts for LLM Tests

5. Test Reports and Coverage

Overview

JUnit XML: The standard test report format

Coverage reports with pytest-cov

Initial setup

Generating coverage in the terminal

Multiple formats at once

Configuration in pyproject.toml

Coverage thresholds: --cov-fail-under

Choosing the right threshold

Coverage in the AI context: The important nuance

Complete workflow: Reports and coverage in GitHub Actions

Artifacts: Saving and downloading reports

Comparisons

--cov-report formats

JUnit XML vs terminal output

Troubleshooting

1. "No data was collected" — Empty coverage report

2. Coverage differs between local and CI

3. Upload artifact fails with "No files were found"

Exercises

Exercise 1: Add coverage to an existing workflow

Exercise 2: Configure pyproject.toml for coverage

Exercise 3: Interpret a coverage report

Exercise 4: Coverage + Job Summary

Summary

Additional resources

Coverage thresholds: `--cov-fail-under`

`--cov-report` formats