4. Linting and Type Checking in CI

Overview

Linting and type checking are standard checks in any CI pipeline. But in AI projects they have their own quirks: how do you configure mypy so it doesn't complain about openai, langchain, or tiktoken, which have no type stubs? Which ruff rules are relevant for projects with HTTP calls and JSON? Why does something pass locally but fail in CI?

This capsule is not "install ruff and mypy." This is: configure both tools for a real AI project with dependencies that don't always cooperate with the type system, integrate them as CI steps that block the merge, and handle the differences between your machine and the runner.

Ruff in CI

Why ruff?

Ruff replaces flake8, isort, pycodestyle, and other tools with a single command that is 10-100x faster. In CI, speed matters: a linter that takes 30 seconds on 1,000 files isn't viable. Ruff does it in under 1 second.

Installation and basic usage

pip install ruff

ruff check src/ tests/ scripts/          # Check for errors
ruff check src/ tests/ scripts/ --fix    # Fix automatically
ruff format src/ tests/ scripts/          # Format the code

Output with errors:

src/ai_app/chain.py:3:1: F401 [*] `os` imported but unused
src/ai_app/chain.py:15:5: E722 Do not use bare `except`
src/ai_app/utils.py:8:1: I001 [*] Import block is un-sorted or un-formatted
Found 3 errors.
[*] 2 fixable with the `--fix` option.

Configuration in pyproject.toml

[tool.ruff]
target-version = "py312"
line-length = 100

[tool.ruff.lint]
select = [
    "E",    # pycodestyle errors
    "W",    # pycodestyle warnings
    "F",    # pyflakes
    "I",    # isort
    "N",    # pep8-naming
    "UP",   # pyupgrade
    "B",    # flake8-bugbear
    "SIM",  # flake8-simplify
    "T20",  # flake8-print (detects print() in production)
    "RET",  # flake8-return
    "PTH",  # flake8-use-pathlib
]

ignore = [
    "E501",   # line too long (the formatter handles it)
    "T201",   # print found (allowed in CLI scripts)
]

[tool.ruff.lint.per-file-ignores]
"scripts/*" = ["T201"]
"tests/*" = ["S101"]

[tool.ruff.lint.isort]
known-first-party = ["src"]

Rules relevant to AI projects

Ruff in CI

      - name: Lint with ruff
        run: ruff check src/ tests/ scripts/

      - name: Check formatting
        run: ruff format --check src/ tests/ scripts/

--check verifies the formatting without modifying files. If there are differences, it fails — forcing the developer to run ruff format locally.

mypy in CI

Why mypy in AI projects?

Python is dynamically typed. In AI projects, types are especially important because:

1. The APIs return complex objects: ChatCompletion, Embedding, Message — without types, you don't know which attributes exist
2. The pipelines have multiple steps: A type error in step 3 isn't detected until runtime
3. The data flows as dicts and JSON: Without types, it's easy to access a key that doesn't exist

Example: mypy catches a real bug

def get_response(client: OpenAI, prompt: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
    )
    return response.choices[0].message.content

src/ai_app/chain.py:8: error: Incompatible return value type
    (got "str | None", expected "str")  [return-value]

mypy detected that .content can be None, but the function declares that it returns str:

def get_response(client: OpenAI, prompt: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
    )
    content = response.choices[0].message.content
    if content is None:
        return ""
    return content

The problem: libraries without type stubs

Many AI libraries don't have complete type stubs. Without configuration, mypy fails on every import of openai, langchain, tiktoken, etc.

mypy configuration for AI projects

[tool.mypy]
python_version = "3.12"
warn_return_any = true
warn_unused_configs = true
disallow_untyped_defs = true
check_untyped_defs = true

[[tool.mypy.overrides]]
module = [
    "openai.*",
    "tiktoken.*",
    "langchain.*",
    "langchain_openai.*",
    "langchain_community.*",
    "chromadb.*",
    "pinecone.*",
    "sentence_transformers.*",
    "transformers.*",
    "anthropic.*",
]
ignore_missing_imports = true

If you add a new AI library, add it to the overrides list.

      - name: Type check with mypy
        run: mypy src/ scripts/

The gap: Local vs CI

You run ruff check and mypy locally — all green. You push, and CI fails. The most common causes:

Cause 1: Different tool versions

Solution: Pin exact versions in requirements-dev.txt:

ruff==0.9.2
mypy==1.14.1

Cause 2: Uncommitted configuration

Solution: Always commit pyproject.toml:

git add pyproject.toml
git commit -m "Add mypy and ruff config"

Cause 3: Missing type dependencies in CI

Solution: Add type stubs to requirements-dev.txt:

mypy==1.14.1
types-requests==2.31.0
types-pyyaml==6.0.12

The definitive test: simulate CI locally

python -m venv .venv-ci-test
source .venv-ci-test/bin/activate
pip install -r requirements-dev.txt
ruff check src/ tests/ scripts/
mypy src/ scripts/
deactivate
rm -rf .venv-ci-test

If it passes here, it passes in CI.

Quality gates: Blocking the merge

A quality gate is a check that must pass for a PR to be mergeable. Configure it on GitHub:

Settings → Branches → Branch protection rules → Add rule
Branch name pattern: main

✅ Require a pull request before merging
✅ Require status checks to pass before merging
   → Search and add: "AI Quality Gate"
✅ Require branches to be up to date before merging

Without branch protection, the checks are informational — you see a ❌ but you can still merge. With branch protection, the merge is blocked until every check passes.

The complete workflow

# .github/workflows/ai-quality-gate.yml
name: AI Quality Gate
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  quality-gate:
    runs-on: ubuntu-latest
    timeout-minutes: 15

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Setup Python 3.12
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements-dev.txt

      # Steps ordered cheapest-first: if ruff fails in <1s, you skip mypy, tests, and API calls entirely
      - name: Lint with ruff
        run: ruff check src/ tests/ scripts/

      - name: Check formatting
        run: ruff format --check src/ tests/ scripts/

      - name: Type check with mypy
        run: mypy src/ scripts/

      - name: Run unit tests
        run: pytest tests/ -v --tb=short --junitxml=test-report.xml
        env:
          PYTHONPATH: ${{ github.workspace }}

      - name: Cost estimation
        # Runs before prompt regression — catches cost spikes without spending API credits
        run: python scripts/estimate_costs.py
        env:
          COST_INCREASE_THRESHOLD_PCT: "20"

      - name: Download prompt baseline
        uses: actions/download-artifact@v4
        with:
          name: prompt-eval-baseline
          path: .
        continue-on-error: true

      - name: Prompt regression testing
        run: python scripts/evaluate_prompts.py
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          PYTHONPATH: ${{ github.workspace }}

      - name: Upload artifacts
        uses: actions/upload-artifact@v4
        # if: always() ensures reports are saved even when checks fail — critical for post-mortem
        if: always()
        with:
          name: ci-reports
          path: |
            test-report.xml
            cost_report.json
            prompt_eval_report.json

      - name: Update baseline on main
        if: github.ref == 'refs/heads/main' && success()
        uses: actions/upload-artifact@v4
        with:
          name: prompt-eval-baseline
          path: prompt_eval_report.json

The order of the steps (deliberate)

1. ruff check        → Free, < 1s      → Style errors
2. ruff format       → Free, < 1s      → Incorrect formatting
3. mypy              → Free, < 5s      → Type errors
4. pytest            → Free, < 30s     → Logic errors
5. Cost estimation   → Free, < 2s      → Cost increases
6. Prompt regression → ~$0.10, < 60s   → Quality degradation

The free and fast checks go first. If ruff fails in 0.5 seconds, you don't spend $0.10 on prompt regression.

Troubleshooting

"mypy fails with 'Library stubs not installed for X'"

Cause: A library has no type stubs and isn't in the overrides list.

Solution: Add the library to [[tool.mypy.overrides]] in pyproject.toml.

"ruff and mypy aren't on the PATH in CI"

Cause: They're not in requirements-dev.txt.

Solution:

# requirements-dev.txt
-r requirements.txt
pytest==8.3.4
ruff==0.9.2
mypy==1.14.1

"ruff format fails but the code looks fine locally"

Cause: Your editor formats with black, but CI uses ruff format.

Solution: Configure your editor to use ruff format:

{
  "[python]": {
    "editor.defaultFormatter": "charliermarsh.ruff",
    "editor.formatOnSave": true
  }
}

Exercises

Exercise 1: Configure a complete pyproject.toml

Create a pyproject.toml with ruff and mypy configuration for an AI project that uses openai, langchain, tiktoken, and pytest.

[project]
name = "my-ai-app"
version = "0.1.0"
requires-python = ">=3.11"

[tool.pytest.ini_options]
testpaths = ["tests"]
pythonpath = ["."]
addopts = "-v --tb=short"

[tool.ruff]
target-version = "py312"
line-length = 100

[tool.ruff.lint]
select = [
    "E", "W", "F", "I", "N", "UP", "B", "SIM", "T20", "RET", "PTH", "PGH",
]
ignore = ["E501"]

[tool.ruff.lint.per-file-ignores]
"scripts/*" = ["T201"]
"tests/*" = ["S101"]

[tool.ruff.lint.isort]
known-first-party = ["src"]

[tool.mypy]
python_version = "3.12"
warn_return_any = true
warn_unused_configs = true
disallow_untyped_defs = true
check_untyped_defs = true

[[tool.mypy.overrides]]
module = [
    "openai.*",
    "tiktoken.*",
    "langchain.*",
    "langchain_openai.*",
    "langchain_community.*",
    "langchain_core.*",
    "chromadb.*",
    "anthropic.*",
]
ignore_missing_imports = true

Exercise 2: Diagnose mypy errors in AI code

This code has 3 errors that mypy detects. Find them and fix them:

from openai import OpenAI


def process_query(query, client):
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": query}],
    )
    content = response.choices[0].message.content
    tokens = response.usage.total_tokens

    return {
        "content": content.strip(),
        "tokens": tokens,
        "cost": tokens * 0.00000015,
    }

from openai import OpenAI


def process_query(query: str, client: OpenAI) -> dict[str, str | int | float]:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": query}],
    )

    content = response.choices[0].message.content
    if content is None:
        content = ""

    tokens = 0
    if response.usage is not None:
        tokens = response.usage.total_tokens

    return {
        "content": content.strip(),
        "tokens": tokens,
        "cost": tokens * 0.00000015,
    }

Exercise 3: A workflow with lint and tests in parallel jobs

Create a workflow with two parallel jobs: one for lint+types and one for tests. Both must pass for the workflow to succeed.

name: CI Pipeline
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  lint-and-types:
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
      - run: pip install ruff mypy
      - name: Lint
        run: ruff check src/ tests/ scripts/
      - name: Format check
        run: ruff format --check src/ tests/ scripts/
      - name: Type check
        run: mypy src/ scripts/

  tests:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
      - run: pip install -r requirements-dev.txt
      - name: Run tests
        run: pytest tests/ -v --tb=short
        env:
          PYTHONPATH: ${{ github.workspace }}

Exercise 4: Caching mypy in CI

mypy's first run is slow. Add caching so subsequent runs are faster.

      - name: Cache mypy results
        uses: actions/cache@v4
        with:
          path: .mypy_cache
          key: mypy-${{ runner.os }}-py3.12-${{ hashFiles('pyproject.toml', 'requirements*.txt') }}
          restore-keys: |
            mypy-${{ runner.os }}-py3.12-

      - name: Type check with mypy
        run: mypy src/ scripts/

Summary

• ✅ ruff replaces multiple linters with a single ultra-fast command — ideal for CI
• ✅ mypy catches type errors that would only show up at runtime — critical for AI projects
• ✅ The mypy overrides configuration is mandatory: libraries without type stubs need ignore_missing_imports
• ✅ pyproject.toml centralizes the ruff, mypy, and pytest configuration
• ✅ The local/CI gap is caused by different versions, uncommitted config, or missing dependencies
• ✅ Quality gates with branch protection turn informational checks into mandatory ones
• ✅ Order matters: free and fast checks go before checks that cost money
• ✅ Caching mypy speeds up subsequent runs from 15s to 3s

Additional resources

1. ruff Documentation — Complete documentation with all the rules
2. mypy Documentation — Configuration guide and advanced usage
3. mypy — Missing Imports — How to handle libraries without stubs
4. GitHub Branch Protection Rules — Configuring quality gates
5. ruff Rules Reference — Complete list of rules with examples