Capsule 06: Observability and Deployment

Capsule description

Your RAG system works locally. Now you'll package it for reproducible execution and add base telemetry to operate with confidence in any environment.

In this capsule you'll implement:

• Multi-stage Dockerfile for the RAG API
• docker-compose.yml with rag-api and chromadb-server
• Environment variables for sensitive configuration
• Health checks in the API and containers
• Prometheus integration for metrics
• Structured logging with a trace_id for traceability

At the end you'll be able to bring up the whole system with a single command and monitor latency, errors, and query volume.

---

Why Reproducible Deployment Matters

The "works on my machine" problem

Developer A: Python 3.11, ChromaDB 0.4.22, Ubuntu
Developer B: Python 3.9, ChromaDB 0.3.x, macOS
Staging: Docker, but without ChromaDB as a service
Production: ???

Without consistent packaging, each environment is a lottery. Docker and docker-compose give you a single source of truth to run the system.

Why observability from day one

In production you need to answer:

• Is the system alive? → Health checks
• How many questions arrive? → Counters
• How slow is /ask? → p95 latency
• Where did this request fail? → Logs with a trace_id

Without this, debugging in production is guessing.

---

Deployment Architecture

                    ┌─────────────────────────────────────────┐
                    │           docker-compose                 │
                    │                                           │
   Client           │  ┌─────────────────┐  ┌──────────────┐  │
   HTTP ────────────┼─►│   rag-api       │  │ chromadb-   │  │
                    │  │   (FastAPI)     │──►│ server      │  │
                    │  │   :8000         │  │ :8001       │  │
                    │  └────────┬────────┘  └──────────────┘  │
                    │           │                              │
                    │           ▼                              │
                    │  ┌─────────────────┐                    │
                    │  │   Volume        │  (persistence)     │
                    │  │   chroma_data   │                    │
                    │  └─────────────────┘                    │
                    └─────────────────────────────────────────┘

---

Multi-Stage Dockerfile

Goal: a small, reproducible image, without build artifacts.

# Dockerfile
# ========== Stage 1: Builder ==========
FROM python:3.11-slim as builder

WORKDIR /app

# Install build dependencies only for build
RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential \
    && rm -rf /var/lib/apt/lists/*

# Create a virtualenv and install dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir --user -r requirements.txt

# ========== Stage 2: Runtime ==========
FROM python:3.11-slim

WORKDIR /app

# Copy only what's needed from the builder
COPY --from=builder /root/.local /root/.local
ENV PATH=/root/.local/bin:$PATH

# Non-root user for security
RUN useradd -m appuser && chown -R appuser:appuser /app
USER appuser

# Application code
COPY --chown=appuser:appuser app/ ./app/
COPY --chown=appuser:appuser main.py .

# Default variables (override with env)
ENV HOST=0.0.0.0
ENV PORT=8000

EXPOSE 8000

# Healthcheck
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
    CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')" || exit 1

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

Explanation of key instructions

Instruction	Purpose
FROM python:3.11-slim	Minimal base image (~50MB vs ~900MB for full)
as builder	Intermediate stage, not included in the final image
COPY --from=builder	Only bins and libs, no compilers
USER appuser	Don't run as root inside the container
HEALTHCHECK	Docker/K8s can detect whether the process is alive
EXPOSE 8000	Documents the port (doesn't open it; run/compose does)

---

docker-compose.yml

# docker-compose.yml
version: "3.8"

services:
  # ========== RAG API ==========
  rag-api:
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - "8000:8000"
    environment:
      - CHROMA_HOST=chromadb
      - CHROMA_PORT=8001
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - LOG_LEVEL=info
    depends_on:
      chromadb:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 10s
    restart: unless-stopped

  # ========== ChromaDB Server ==========
  chromadb:
    image: chromadb/chroma:latest
    ports:
      - "8001:8000"
    volumes:
      - chroma_data:/chroma/chroma
    environment:
      - IS_PERSISTENT=TRUE
      - ANONYMIZED_TELEMETRY=FALSE
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/api/v1/heartbeat"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 5s
    restart: unless-stopped

volumes:
  chroma_data:

Environment variables

Create .env (don't commit, use only locally):

# .env.example (rename to .env and fill in)
OPENAI_API_KEY=sk-...
LOG_LEVEL=info
CHROMA_HOST=chromadb
CHROMA_PORT=8001

# Bring everything up
docker-compose up -d

# View logs
docker-compose logs -f rag-api

# Stop and remove volumes
docker-compose down -v

---

Environment Variables in the Application

# app/config.py
import os
from pydantic_settings import BaseSettings

class Settings(BaseSettings):
    chroma_host: str = os.getenv("CHROMA_HOST", "localhost")
    chroma_port: int = int(os.getenv("CHROMA_PORT", "8001"))
    openai_api_key: str = os.getenv("OPENAI_API_KEY", "")
    log_level: str = os.getenv("LOG_LEVEL", "info")
    
    class Config:
        env_file = ".env"

settings = Settings()

Centralize all configuration here. Don't hardcode URLs or API keys.

---

Health Check Endpoint

# app/main.py (fragment)
from fastapi import FastAPI, Request
import httpx

app = FastAPI(title="RAG API", version="1.0.0")

@app.get("/health")
async def health(request: Request):
    """Health check: API + dependencies."""
    status = {"status": "ok", "version": "1.0.0"}
    try:
        chroma_url = f"http://{settings.chroma_host}:{settings.chroma_port}/api/v1/heartbeat"
        async with httpx.AsyncClient() as client:
            r = await client.get(chroma_url, timeout=2.0)
            status["chromadb"] = "ok" if r.status_code == 200 else "degraded"
    except Exception as e:
        status["chromadb"] = "error"
        status["chromadb_error"] = str(e)
    return status

Kubernetes, Render, Railway, and similar use /health to know if the service is ready.

---

Prometheus Integration

Dependencies

# requirements.txt (add)
prometheus-client>=0.19.0

Minimal metrics

# app/metrics.py
from prometheus_client import Counter, Histogram, generate_latest, CONTENT_TYPE_LATEST
from fastapi import Response

# Counters
request_count = Counter(
    "rag_requests_total",
    "Total requests",
    ["endpoint", "method", "status"]
)
ask_latency = Histogram(
    "rag_ask_latency_seconds",
    "Latency of /ask endpoint",
    buckets=[0.5, 1.0, 1.5, 2.0, 3.0, 5.0]
)
search_latency = Histogram(
    "rag_search_latency_seconds",
    "Latency of /search endpoint",
    buckets=[0.05, 0.1, 0.2, 0.5, 1.0]
)

def get_metrics():
    return generate_latest()

@app.get("/metrics")
def metrics():
    return Response(
        content=get_metrics(),
        media_type=CONTENT_TYPE_LATEST
    )

Use in endpoints

import time
from app.metrics import request_count, ask_latency

@app.post("/ask")
async def ask(payload: dict, request: Request):
    trace_id = request.headers.get("X-Trace-ID", "unknown")
    start = time.perf_counter()
    try:
        result = await do_ask(payload["question"], trace_id)
        request_count.labels(endpoint="/ask", method="POST", status="200").inc()
        ask_latency.observe(time.perf_counter() - start)
        return result
    except Exception as e:
        request_count.labels(endpoint="/ask", method="POST", status="500").inc()
        raise

---

Structured Logging

# app/logging_config.py
import logging
import json
from datetime import datetime

class StructuredFormatter(logging.Formatter):
    def format(self, record):
        log_obj = {
            "timestamp": datetime.utcnow().isoformat() + "Z",
            "level": record.levelname,
            "message": record.getMessage(),
            "module": record.module,
        }
        if hasattr(record, "trace_id"):
            log_obj["trace_id"] = record.trace_id
        if record.exc_info:
            log_obj["exception"] = self.formatException(record.exc_info)
        return json.dumps(log_obj, ensure_ascii=False)

def setup_logging():
    handler = logging.StreamHandler()
    handler.setFormatter(StructuredFormatter())
    root = logging.getLogger()
    root.addHandler(handler)
    root.setLevel(logging.INFO)

# Use in an endpoint
logger = logging.getLogger(__name__)

async def do_ask(question: str, trace_id: str):
    logger.info("Processing ask", extra={"trace_id": trace_id, "question_len": len(question)})
    # ...

Example output:

{"timestamp": "2026-03-13T12:00:00.000Z", "level": "INFO", "message": "Processing ask", "trace_id": "req-abc123", "question_len": 45}

---

Deployment Checklist

• It can be brought up with docker-compose up -d
• The rag-api healthcheck responds OK
• The ChromaDB healthcheck responds OK
• Structured logs (JSON) enabled
• /metrics exposes latency and counters
• Sensitive variables in .env (not in the repo)
• ChromaDB persistence in a named volume

---

Recommended Deployment Flow

1. Build a reproducible image
   docker build -t rag-api:latest .

2. Validate in staging
   docker-compose -f docker-compose.yml up
   curl http://localhost:8000/health
   curl -X POST http://localhost:8000/ask -d '{"question":"test"}'

3. Smoke test critical endpoints
   - GET /health
   - GET /search?q=test
   - POST /ask with a known question

4. Promotion to production
   - Image tag: rag-api:v1.2.3
   - Deploy on a platform (Render, Railway, K8s)
   - Verify metrics on a dashboard

---

Minimal Post-Deploy Metrics

Metric	Where	Alert threshold
/health availability	Prometheus / uptime	< 99%
p95 /ask	Prometheus histogram	> 2.5s
Error rate per endpoint	Prometheus counter	> 1%
Query volume/min	Prometheus counter	— (observation only)
ChromaDB heartbeat	Health check	error

---

Exercises with Detailed Solutions

Exercise 1: Add a readiness endpoint

Goal: /ready that verifies ChromaDB + connection to OpenAI (without calling, just connect).

Solution:

@app.get("/ready")
async def ready():
    checks = {}
    try:
        async with httpx.AsyncClient() as c:
            r = await c.get(f"http://{settings.chroma_host}:{settings.chroma_port}/api/v1/heartbeat", timeout=2.0)
        checks["chromadb"] = r.status_code == 200
    except Exception:
        checks["chromadb"] = False
    checks["openai_configured"] = bool(settings.openai_api_key)
    all_ok = all(checks.values())
    return JSONResponse(
        status_code=200 if all_ok else 503,
        content={"ready": all_ok, "checks": checks}
    )

Exercise 2: Cost-per-query metric

Goal: Expose rag_cost_per_query (Gauge or Counter) estimating tokens used.

Solution:

from prometheus_client import Gauge
cost_per_query = Gauge("rag_estimated_cost_usd", "Estimated cost per query in USD")

# In do_ask, after calling OpenAI:
# cost_per_query.set( (prompt_tokens * 0.001 + completion_tokens * 0.002) / 1000 )  # approx

Exercise 3: Dockerfile for development (hot reload)

Goal: Dockerfile.dev that mounts the code and uses uvicorn --reload.

Solution:

FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000", "--reload"]

In docker-compose.yml add a rag-api-dev service with build: Dockerfile.dev and volumes: ["./app:/app/app"].

Exercise 4: Logs with a trace_id in every request

Goal: A middleware that injects a trace_id into each request and adds it to the logs.

Solution:

import uuid
from starlette.middleware.base import BaseHTTPMiddleware

class TraceMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request, call_next):
        trace_id = request.headers.get("X-Trace-ID") or str(uuid.uuid4())[:8]
        request.state.trace_id = trace_id
        response = await call_next(request)
        response.headers["X-Trace-ID"] = trace_id
        return response

app.add_middleware(TraceMiddleware)

Exercise 5: Health check that validates an existing collection

Goal: /health must check that the default collection exists and has documents.

Solution:

@app.get("/health")
async def health():
    try:
        client = get_chroma_client()
        col = client.get_collection("rag_docs")
        count = col.count()
        return {"status": "ok", "chromadb": "ok", "doc_count": count}
    except Exception as e:
        return JSONResponse(
            status_code=503,
            content={"status": "degraded", "chromadb": str(e)}
        )

Exercise 6: docker-compose with Redis for cache (optional)

Goal: Add a redis service and a REDIS_URL variable for the API.

Solution:

  redis:
    image: redis:7-alpine
    ports: ["6379:6379"]
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s

  rag-api:
    environment:
      - REDIS_URL=redis://redis:6379/0
    depends_on:
      redis:
        condition: service_healthy

---

Deployment Troubleshooting

"Works locally, fails in the container"

Check environment variables: CHROMA_HOST must be chromadb (the service name), not localhost. Check persistence paths: ChromaDB must write to a volume, not to ephemeral /tmp.

"Healthcheck OK, but /ask fails"

Validate the ChromaDB dependency: does the collection exist? Does it have data? Verify OPENAI_API_KEY and that the container has internet access for the OpenAI API.

"We don't see useful logs"

Standardize the JSON format and add a trace_id to each log. Use LOG_LEVEL=DEBUG in development and INFO in production.

"The image is too heavy"

Use multi-stage and slim. Avoid apt-get install of heavy packages. Review with docker history rag-api:latest.

"ChromaDB loses data on restart"

Make sure the volume is mounted correctly. In docker-compose, chroma_data must map to the path ChromaDB uses to persist (by default /chroma/chroma in the official image).

---

Quick Reference Commands

# Bring everything up
docker-compose up -d

# View status
docker-compose ps
docker-compose logs -f rag-api

# Rebuild after changes
docker-compose build --no-cache rag-api
docker-compose up -d rag-api

# Run tests against the running API
API_BASE_URL=http://localhost:8000 pytest tests/integration -v

# Scale replicas (if configured)
docker-compose up -d --scale rag-api=2

---

Full Structured Log Example

{
  "timestamp": "2026-03-13T12:00:00.123Z",
  "level": "INFO",
  "message": "Processing ask request",
  "trace_id": "req-a1b2c3d4",
  "module": "ask",
  "question_length": 42,
  "retrieval_docs_count": 5,
  "latency_ms": 1850
}

This format lets you search by trace_id in any log aggregator (Elasticsearch, Datadog, etc.).

---

Suggested Grafana Dashboard

If you use Prometheus + Grafana, create a dashboard with:

1. Latency panel: graph of histogram_quantile(0.95, rag_ask_latency_seconds_bucket)
2. Throughput panel: rate(rag_requests_total[5m])
3. Errors panel: rate(rag_requests_total{status="500"}[5m]) / rate(rag_requests_total[5m])
4. Health panel: up/down of the target

---

Summary

• You packaged the system with a multi-stage Dockerfile and docker-compose (rag-api + chromadb-server).
• You configured environment variables for ChromaDB, OpenAI, and the log level.
• You implemented health checks in the API and ChromaDB for failure detection.
• You integrated Prometheus with latency metrics and counters.
• You configured structured logging with a trace_id.
• The system comes up with a single command and is observable from day one.

Next step: Final hardening (errors, cache, security, documentation) in Capsule 07.

---

Additional Resources

• Docker Documentation — Dockerfile, multi-stage
• Docker Compose — service orchestration
• Prometheus — metrics and alerting
• prometheus-client Python — instrumentation
• ChromaDB Docker — ChromaDB deployment
• FastAPI Deployment — deploy options
• 12-Factor App — configuration, logs, processes
• OpenTelemetry — traces for advanced observability

---

Estimated time: 50-60 minutes
Next: 07-hardening-final.md