Module 2: Local & Container Deployment

4. Health Checks and Dependencies

Next →

Overview

In this capsule you'll implement health checks that verify each service's real readiness and dependency ordering that ensures your containers start in the right order. By the end, your Docker Compose won't accept traffic until ALL services are truly operational — not just "running."

Context: A "running" service isn't the same as a "ready" service. Redis may be starting but not accepting connections. Your API may be importing libraries but not responding to requests. Health checks verify real readiness; dependency ordering ensures the API doesn't try to connect to Redis before Redis is ready.

Health Checks: More than "the process exists"

The problem without health checks

# Without health checks, Compose only verifies that the process started
docker compose up -d
# ✔ cache-1  Started     # But does it accept connections?
# ✔ api-1    Started     # But does /health respond 200?

# The API tries to connect to Redis, which is still initializing
# → ConnectionRefusedError
# → The API crashes or returns 500s during the first few seconds

Health check anatomy

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
  interval: 30s      # How often to check
  timeout: 10s       # Maximum wait time per check
  retries: 3         # Attempts before declaring unhealthy
  start_period: 10s  # Initial grace (doesn't count as a failure)
Timeline of a health check:
0s ─────────── start_period (10s) ──────────── 10s
                                                │
10s ── check ── 40s ── check ── 70s ── check ── 100s
        │              │              │
      pass? ─── YES → healthy    NO → retry (up to 3)
                                      │
                                   3 failures → unhealthy

Health checks per service

services:
  cache:
    image: redis:7-alpine
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 5s
      retries: 3
    # Redis responds PONG when it's ready to accept commands

  api:
    build: ./api
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 15s  # More grace: the API needs to import libraries
    # The /health endpoint verifies that the API AND Redis are operational

  vectordb:
    image: qdrant/qdrant:latest
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:6333/healthz"]
      interval: 15s
      timeout: 5s
      retries: 3

Health endpoint in the API

# api/main.py — Health check that verifies dependencies
@app.get("/health")
def health():
    checks = {"api": "up"}
    overall = "healthy"

    # Check Redis
    try:
        cache.ping()
        checks["redis"] = "up"
    except Exception:
        checks["redis"] = "down"
        overall = "degraded"

    status_code = 200 if overall == "healthy" else 503

    from fastapi.responses import JSONResponse
    return JSONResponse(
        status_code=status_code,
        content={"status": overall, "services": checks}
    )

Readiness vs Liveness: Two Types of Health Check

In production systems, a single health check isn't enough. There are two different questions you need to answer:

  • Liveness: "Is the process alive?" → If not, restart it.
  • Readiness: "Can the service accept traffic?" → If not, don't send it requests.

The difference in practice

Scenario: Your API is importing a large ML model (30 seconds)

Liveness:  ✅ The process is alive (it would respond to a basic ping)
Readiness: ❌ It's not ready to accept requests (the model hasn't loaded)

Correct action: Do NOT restart (it's alive), do NOT send traffic (it's not ready)

Implement both in FastAPI

# api/main.py
from fastapi import FastAPI
from contextlib import asynccontextmanager
import redis
import time

app_ready = False
startup_time = None


@asynccontextmanager
async def lifespan(app: FastAPI):
    global app_ready, startup_time
    startup_time = time.time()

    # Simulate heavy load: import model, warm cache, etc.
    await initialize_services()
    app_ready = True

    yield

    app_ready = False


app = FastAPI(lifespan=lifespan)


@app.get("/health/live")
def liveness():
    """Is the process alive? Basic check, no external dependencies."""
    return {
        "status": "alive",
        "uptime_seconds": round(time.time() - startup_time, 1),
    }


@app.get("/health/ready")
def readiness():
    """Can it accept traffic? Verifies critical dependencies."""
    from fastapi.responses import JSONResponse

    if not app_ready:
        return JSONResponse(
            status_code=503,
            content={"status": "not_ready", "reason": "still initializing"},
        )

    checks = {}
    all_healthy = True

    try:
        cache.ping()
        checks["redis"] = "up"
    except Exception:
        checks["redis"] = "down"
        all_healthy = False

    try:
        import urllib.request
        urllib.request.urlopen("http://vectordb:6333/healthz", timeout=2)
        checks["vectordb"] = "up"
    except Exception:
        checks["vectordb"] = "down"
        all_healthy = False

    status_code = 200 if all_healthy else 503
    status = "ready" if all_healthy else "degraded"

    return JSONResponse(
        status_code=status_code,
        content={"status": status, "checks": checks},
    )

Health checks in Docker Compose: map to readiness

services:
  api:
    build: ./api
    healthcheck:
      # Use the readiness endpoint, not the liveness one
      test: ["CMD", "curl", "-f", "http://localhost:8000/health/ready"]
      interval: 15s
      timeout: 10s
      retries: 3
      start_period: 30s

Docker Compose has only one health check (it doesn't distinguish liveness/readiness like Kubernetes). Use the readiness endpoint — it's the one that matters for dependency ordering.

Health Check Strategies for AI Services

AI services have unique health check needs. A normal API only needs to verify "can I respond over HTTP?" AI services need to verify loaded models, LLM connectivity, and vector store states.

Verify LLM connectivity

# api/health.py
from openai import OpenAI
import time


def check_llm_connectivity(client: OpenAI, timeout: float = 5.0) -> dict:
    """
    Verifies that the OpenAI API responds.
    Doesn't send a full prompt — uses a minimal request.
    """
    start = time.time()
    try:
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": "1"}],
            max_tokens=1,
        )
        latency_ms = round((time.time() - start) * 1000)
        return {
            "status": "up",
            "latency_ms": latency_ms,
            "model": "gpt-4o-mini",
        }
    except Exception as e:
        return {"status": "down", "error": type(e).__name__}

Verify vector store readiness

# api/health.py
import urllib.request
import json


def check_vector_store(qdrant_url: str = "http://vectordb:6333") -> dict:
    """Verifies that Qdrant responds and has collections."""
    try:
        response = urllib.request.urlopen(
            f"{qdrant_url}/collections", timeout=3
        )
        data = json.loads(response.read())
        collections = [c["name"] for c in data.get("result", {}).get("collections", [])]
        return {
            "status": "up",
            "collections": collections,
            "collection_count": len(collections),
        }
    except Exception as e:
        return {"status": "down", "error": str(e)}

Complete health endpoint for an AI app

# api/main.py
from health import check_llm_connectivity, check_vector_store
from config import settings


@app.get("/health/detailed")
def health_detailed():
    """Complete health check with latencies and the state of each dependency."""
    checks = {}

    # Redis
    import time
    start = time.time()
    try:
        cache.ping()
        checks["redis"] = {
            "status": "up",
            "latency_ms": round((time.time() - start) * 1000),
        }
    except Exception as e:
        checks["redis"] = {"status": "down", "error": str(e)}

    # LLM (only in readiness, not in every basic health check)
    checks["llm"] = check_llm_connectivity(openai_client)

    # Vector Store
    checks["vectordb"] = check_vector_store()

    # Overall status
    statuses = [c["status"] for c in checks.values()]
    if all(s == "up" for s in statuses):
        overall = "healthy"
    elif checks["redis"]["status"] == "up":
        overall = "degraded"
    else:
        overall = "unhealthy"

    from fastapi.responses import JSONResponse
    return JSONResponse(
        status_code=200 if overall == "healthy" else 503,
        content={"status": overall, "checks": checks},
    )

An important detail: the health check Docker Compose uses (/health/ready) must be fast (< 2 seconds). The /health/detailed endpoint with LLM latencies is for monitoring, not for the Compose health check.

Dependency Ordering

depends_on with a condition

services:
  api:
    depends_on:
      cache:
        condition: service_healthy  # Wait for cache to be HEALTHY
      vectordb:
        condition: service_healthy

  cache:
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]

  vectordb:
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:6333/healthz"]
Startup order:
1. cache + vectordb start in parallel
2. Compose waits for both to report "healthy"
3. api starts ONLY when cache AND vectordb are healthy
4. If cache fails the healthcheck 3 times → api does NOT start

depends_on without a condition (insufficient)

# ❌ INSUFFICIENT: only waits for the container to start, not to be ready
services:
  api:
    depends_on:
      - cache  # Only verifies that cache "started", not that it's healthy

Complex Scenario: Multi-Service AI Pipeline

In real AI apps, services form dependency chains. A typical RAG pipeline has 3 layers:

# docker-compose.yml — RAG pipeline with dependency layers
services:
  # Layer 1: Infrastructure (no dependencies, start in parallel)
  cache:
    image: redis:7-alpine
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 3

  vectordb:
    image: qdrant/qdrant:latest
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:6333/healthz"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 10s

  # Layer 2: Worker that depends on infra
  embeddings-worker:
    build: ./workers/embeddings
    depends_on:
      vectordb:
        condition: service_healthy
      cache:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8001/health')"]
      interval: 15s
      timeout: 10s
      retries: 3
      start_period: 20s

  # Layer 3: Main API (depends on everything)
  api:
    build: ./api
    depends_on:
      cache:
        condition: service_healthy
      vectordb:
        condition: service_healthy
      embeddings-worker:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health/ready"]
      interval: 15s
      timeout: 10s
      retries: 3
      start_period: 30s
Resulting startup order:

Layer 1 (parallel):  cache ──────┐
                     vectordb ───┘
                                 │ both healthy
                                 ▼
Layer 2:             embeddings-worker
                                 │ healthy
                                 ▼
Layer 3:             api

Restart policies

services:
  api:
    restart: unless-stopped
    # Options:
    # no            — never restarts
    # always        — always restarts (even on exit 0)
    # on-failure    — restarts only if exit != 0
    # unless-stopped — restarts except if you stop it

Graceful Shutdown and Signal Handling

When you run docker compose down, Docker sends SIGTERM to each container. If the process doesn't terminate in 10 seconds (default), it sends SIGKILL. For AI apps, those 10 seconds may not be enough — a request to the LLM can take 30 seconds.

The problem

docker compose down
  → SIGTERM to all containers
  → API is processing an LLM request (takes 15s)
  → 10 seconds... Docker sends SIGKILL
  → Request cut off, user gets an error, possible cache corruption

Configure the stop timeout

services:
  api:
    build: ./api
    stop_grace_period: 30s  # Wait 30s before SIGKILL
    restart: unless-stopped

Handle SIGTERM in Python

# api/main.py
import signal
import asyncio
from fastapi import FastAPI
from contextlib import asynccontextmanager

shutdown_event = asyncio.Event()


def handle_sigterm(signum, frame):
    shutdown_event.set()


signal.signal(signal.SIGTERM, handle_sigterm)


@asynccontextmanager
async def lifespan(app: FastAPI):
    yield
    cache.close()


app = FastAPI(lifespan=lifespan)


@app.middleware("http")
async def check_shutdown(request, call_next):
    if shutdown_event.is_set():
        from fastapi.responses import JSONResponse
        return JSONResponse(
            status_code=503,
            content={"error": "Server is shutting down"},
        )
    return await call_next(request)

The pattern: register a handler for SIGTERM that sets a flag. A middleware rejects new requests when the flag is active. The lifespan closes connections at the end.

Shutdown ordering in Compose

docker compose down stops in reverse order of depends_on: first the API (stops receiving traffic), then cache (the API no longer needs it). If you need a specific order: docker compose stop api && docker compose stop cache.

Complete Pattern: Ordering + Health + Restart

services:
  cache:
    image: redis:7-alpine
    command: redis-server --appendonly yes
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 5s
      retries: 3
    restart: unless-stopped

  api:
    build: ./api
    ports:
      - "8000:8000"
    environment:
      - REDIS_URL=redis://cache:6379
    depends_on:
      cache:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 15s
    restart: unless-stopped

Troubleshooting

Problem 1: "The container stays in 'health: starting' forever"

# Check the healthcheck logs
docker inspect --format='{{json .State.Health}}' module-02-api-1 | python -m json.tool

# The start_period may be too short
# If your app takes 20s to start, start_period must be >20s

Problem 2: "curl not found in the container"

# Add curl to the Dockerfile
FROM python:3.11-slim
RUN apt-get update && apt-get install -y curl && rm -rf /var/lib/apt/lists/*

Or use an alternative without curl:

healthcheck:
  test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"]

Problem 3: "api starts before Redis is ready"

Verify that depends_on uses condition: service_healthy (not just depends_on: - cache), and that cache has a healthcheck defined.

Problem 4: "The health check passes but the service doesn't work well"

# Your health check is too simple. If it only verifies that the process exists,
# it doesn't detect problems like: exhausted connection pool, full memory, deadlock.

# Solution: the health check must test the real functionality
# ❌ Only verifies that the port responds
test: ["CMD", "curl", "-f", "http://localhost:8000/"]

# ✅ Verifies that the dependencies work too
test: ["CMD", "curl", "-f", "http://localhost:8000/health/ready"]

# Check the latest health check results:
docker inspect --format='{{range .State.Health.Log}}{{.Output}}{{end}}' module-02-api-1

Problem 5: "Container restarts in a loop (restart loop)"

# The container fails, restarts, fails again, infinitely
docker compose logs api --tail 50
# Look for the error causing the crash

# Common causes:
# 1. Missing environment variable → the app crashes on startup
# 2. Unreachable dependency → the app crashes on connect
# 3. Port already in use inside the container

# Diagnosis:
docker compose ps
# NAME          STATUS                   PORTS
# api-1         restarting (3 seconds)

# To debug without automatic restart:
docker compose run --rm api bash
# Inside the container, run the command manually:
python -c "from config import settings; print(settings)"

Hands-On Exercises

Exercise 1: Multi-service health check

Implement a /health/detailed endpoint that verifies Redis, the LLM model (a test call), and returns response times.

See solution 
import time

@app.get("/health/detailed")
def health_detailed():
    checks = {}

    # Redis check
    start = time.time()
    try:
        cache.ping()
        checks["redis"] = {"status": "up", "latency_ms": round((time.time() - start) * 1000)}
    except Exception as e:
        checks["redis"] = {"status": "down", "error": str(e)}

    # LLM check
    start = time.time()
    try:
        r = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": "ping"}],
            max_tokens=5
        )
        checks["llm"] = {"status": "up", "latency_ms": round((time.time() - start) * 1000)}
    except Exception as e:
        checks["llm"] = {"status": "down", "error": str(e)}

    overall = "healthy" if all(c["status"] == "up" for c in checks.values()) else "degraded"
    return {"status": overall, "checks": checks}

Exercise 2: Graceful degradation

Modify the API so it works (without cache) if Redis goes down, instead of returning 500.

See solution 
@app.post("/ask")
def ask(request: AskRequest):
    # Try cache, but don't fail if Redis is down
    cached = None
    try:
        if request.use_cache:
            cached = cache.get(cache_key)
    except redis.ConnectionError:
        pass  # Cache unavailable, proceed without it

    if cached:
        return AskResponse(answer=json.loads(cached)["answer"], cached=True)

    # LLM call (always works if Redis is down)
    response = client.chat.completions.create(...)

    # Try to cache, but don't fail if Redis is down
    try:
        cache.setex(cache_key, settings.cache_ttl, json.dumps({...}))
    except redis.ConnectionError:
        pass  # Can't cache, but response still works

    return AskResponse(answer=answer, cached=False, tokens_used=tokens)

Exercise 3: Separate readiness and liveness

Implement two separate endpoints: /health/live (only verifies that the process runs) and /health/ready (verifies Redis + vector store). Configure Docker Compose to use the readiness endpoint.

See solution 
# api/main.py
import time

startup_time = time.time()
app_ready = False


@app.get("/health/live")
def liveness():
    return {
        "status": "alive",
        "uptime_seconds": round(time.time() - startup_time, 1),
    }


@app.get("/health/ready")
def readiness():
    from fastapi.responses import JSONResponse

    if not app_ready:
        return JSONResponse(
            status_code=503,
            content={"status": "initializing"},
        )

    checks = {}
    try:
        cache.ping()
        checks["redis"] = "up"
    except Exception:
        checks["redis"] = "down"

    try:
        import urllib.request
        urllib.request.urlopen("http://vectordb:6333/healthz", timeout=2)
        checks["vectordb"] = "up"
    except Exception:
        checks["vectordb"] = "down"

    all_up = all(v == "up" for v in checks.values())
    return JSONResponse(
        status_code=200 if all_up else 503,
        content={"status": "ready" if all_up else "degraded", "checks": checks},
    )
# docker-compose.yml
services:
  api:
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health/ready"]
      interval: 15s
      timeout: 10s
      retries: 3
      start_period: 30s

The practical difference: if Redis goes down temporarily, /health/live still responds 200 (don't restart the container), but /health/ready responds 503 (don't send new traffic).

Exercise 4: Graceful shutdown with stop_grace_period

Configure your API to handle SIGTERM correctly: stop accepting new requests, finish in-flight requests, close connections, and then die cleanly. Configure a stop_grace_period of 30 seconds.

See solution 
# api/main.py
import signal
import asyncio
from fastapi import FastAPI
from contextlib import asynccontextmanager

shutdown_event = asyncio.Event()


def handle_sigterm(signum, frame):
    shutdown_event.set()


signal.signal(signal.SIGTERM, handle_sigterm)


@asynccontextmanager
async def lifespan(app: FastAPI):
    yield
    # Cleanup
    cache.close()
    await asyncio.sleep(0.5)


app = FastAPI(lifespan=lifespan)


@app.middleware("http")
async def reject_during_shutdown(request, call_next):
    if shutdown_event.is_set():
        from fastapi.responses import JSONResponse
        return JSONResponse(
            status_code=503,
            content={"error": "Shutting down, try another instance"},
        )
    return await call_next(request)
# docker-compose.yml
services:
  api:
    build: ./api
    stop_grace_period: 30s
    restart: unless-stopped
# Test graceful shutdown
docker compose down
# Watch the logs: you'll see "SIGTERM received" and then cleanup
docker compose logs api --tail 20

Summary

  • Health checks verify that a service is truly operational, not just that the process exists.
  • Readiness vs liveness: readiness verifies dependencies (can I accept traffic?), liveness verifies the process (am I alive?). Docker Compose uses only one — choose readiness.
  • depends_on with condition: service_healthy ensures correct ordering.
  • start_period gives initial grace so the service starts before failures are counted.
  • Restart policies (unless-stopped) keep services running through crashes.
  • The /health endpoint must verify dependencies (Redis, vector store), not just the API.
  • Graceful degradation: if a dependency fails, the app degrades functionality instead of crashing.
  • Graceful shutdown: handle SIGTERM, use stop_grace_period, close connections cleanly.

Additional Resources

  1. Docker Compose Healthcheck — Official reference
  2. Docker Container Health — Healthcheck in the Dockerfile
  3. Health Check Patterns — Microsoft — Health monitoring patterns
  4. Graceful Degradation Patterns — Degradation patterns
  5. Docker Compose depends_on — Dependency ordering
  6. FastAPI Lifespan Events — Startup and shutdown in FastAPI
  7. Docker Stop Grace Period — Configure the shutdown timeout
  8. Python Signal Handling — Handle SIGTERM in Python
← Previous3. Environment ConfigurationNext →5. Networking and Communication Between Services