2. Stack architecture

Overview

Before you write a single line of Dockerfile or docker-compose.yml, you need to design the system. This capsule defines the complete architecture of the Production AI Stack: which services make it up, what each one is responsible for, how they talk to each other, which ports they expose, what data persists, and which network they operate on. This design is the blueprint you'll implement in the next capsules.

Why it matters: Starting to write a docker-compose.yml without a clear design produces a stack that "works" but that nobody understands. When something breaks at 3 AM, you need to know exactly which service talks to which, over which port, and where the data lives. Documented architecture is your survival map.

Connection with the module: This capsule produces the design. Capsules 03-04 implement it. Capsules 05-07 harden it. Capsule 08 delivers it complete.

The stack: big picture

Three services, one purpose

The Production AI Stack has 3 services that work together to serve a semantic search API with caching:

┌─────────────────────────────────────────────────────────────┐
│                    Production AI Stack                       │
│                                                              │
│   ┌──────────────────┐                                      │
│   │    FastAPI API    │  Main service                        │
│   │    (api)          │  Custom Dockerfile                   │
│   │    Port: 8000     │  Multi-stage, non-root, health check│
│   └────────┬─────────┘                                      │
│            │                                                 │
│     ┌──────┴──────┐                                         │
│     │              │                                         │
│     ▼              ▼                                         │
│   ┌──────────┐  ┌──────────┐                                │
│   │ ChromaDB │  │  Redis   │                                │
│   │ (chroma) │  │  (redis) │                                │
│   │ Port:8000│  │ Port:6379│                                │
│   │ Int: 8100│  │          │                                │
│   └──────────┘  └──────────┘                                │
│                                                              │
│   Network: ai-network (bridge)                              │
│   Volumes: chroma-data, redis-data                          │
└─────────────────────────────────────────────────────────────┘

Why these three services

This trio is the minimum viable pattern of a RAG application: an API that searches a vector database and caches the results.

Service 1: FastAPI API

Responsibilities

FastAPI API (service: api)
├── Receive HTTP requests from users
├── Validate input with Pydantic models
├── Query Redis for cache hits
├── Query ChromaDB for semantic search
├── Process and format results
├── Cache responses in Redis
├── Expose a health check endpoint
└── Produce structured logs (JSON)

Technical specs

Endpoints

GET  /health      → Health check (API + Redis + ChromaDB)
POST /documents   → Insert documents into ChromaDB
GET  /search      → Semantic search in ChromaDB (with Redis cache)
GET  /cache/stats → Redis cache statistics
DELETE /cache     → Clear the cache
GET  /info        → Container info (user, version, platform)

Dockerfile overview

The API's Dockerfile applies every practice from the previous modules:

Builder stage:
├── FROM python:3.11-slim AS builder
├── COPY requirements.txt
└── RUN pip install --prefix=/install

Runtime stage:
├── FROM python:3.11-slim
├── Build metadata (LABEL)
├── Create non-root user (appuser)
├── COPY --from=builder dependencies
├── COPY --chown=appuser:appuser application
├── USER appuser
├── ENV PYTHONUNBUFFERED=1
├── HEALTHCHECK python healthcheck.py
└── CMD uvicorn

The full detail of the Dockerfile is in capsule 03.

Service 2: ChromaDB

Responsibilities

ChromaDB (service: chroma)
├── Store vector embeddings
├── Run semantic search (similarity search)
├── Persist data in a volume
└── Expose an HTTP API for queries

Technical specs

Communication

The API connects to ChromaDB using the service name as the hostname:

import chromadb

client = chromadb.HttpClient(host="chroma", port=8000)

Docker Compose resolves chroma to the container's internal IP on the ai-network network. You don't need hardcoded IPs.

Why you don't write a Dockerfile for ChromaDB

ChromaDB provides an official image that's already optimized and configured. Writing a custom Dockerfile for ChromaDB would mean:

• ❌ Duplicating work the ChromaDB team already did
• ❌ Risking incompatibilities with its internal configuration
• ❌ Taking on the responsibility of maintaining security updates

The rule is: use official images when they exist. Only write custom Dockerfiles when it's your own application.

Service 3: Redis

Responsibilities

Redis (service: redis)
├── Store cached responses (key-value)
├── Handle TTL (time-to-live) for automatic expiration
├── Persist data in a volume (optional, for cache warming)
└── Respond to health checks (PING → PONG)

Technical specs

Communication

The API connects to Redis using the service name:

import redis

r = redis.from_url("redis://redis:6379")

redis://redis:6379 — the first "redis" is the protocol, the second is the service's hostname in Docker Compose.

Redis Alpine vs standard Redis

redis:7          → ~130 MB (Debian-based)
redis:7-alpine   → ~30 MB  (Alpine Linux-based)

For a cache layer, Alpine is the right choice. It's 4x smaller and has the same functionality for standard use.

Communication between services

Network: ai-network

Every service lives on a custom network, ai-network:

networks:
  ai-network:
    driver: bridge

Why a custom network instead of the default one?

• ✅ Automatic DNS: Services resolve by name (redis, chroma, api)
• ✅ Isolation: Only this stack's services can talk to each other
• ✅ Control: You can inspect the network with docker network inspect ai-network

Communication map

┌──────────────────────────────────────────────────┐
│                  ai-network                       │
│                                                   │
│  api ──────────────► chroma                       │
│   │   HTTP :8000     (semantic search)            │
│   │                                               │
│   └────────────────► redis                        │
│       Redis :6379    (cache read/write)            │
│                                                   │
│  chroma ──✗──► redis   (they don't communicate)   │
│  redis  ──✗──► chroma  (they don't communicate)   │
│                                                   │
└──────────────────────────────────────────────────┘

External (host):
  User    ──► api:8000     (HTTP requests)
  Dev    ──► chroma:8100   (direct debug, dev only)
  Dev    ──► redis:6379    (direct debug, dev only)

ChromaDB and Redis never talk to each other. Only the API communicates with both. That simplifies the architecture and the debugging.

Full flow: a search request

Let's follow a complete request through the stack:

1. User sends: POST /search {"query": "machine learning basics"}

2. FastAPI API receives the request
   └── Generates a cache key: "search:hash(machine learning basics)"

3. API queries Redis: GET search:abc123
   ├── Cache HIT  → Returns the cached response (step 7)
   └── Cache MISS → Continues to step 4

4. API queries ChromaDB: query(collection="documents",
                                 query_texts=["machine learning basics"],
                                 n_results=5)
   └── ChromaDB returns the 5 most similar documents

5. API processes the results:
   └── Formats, filters, and sorts the documents

6. API writes to Redis: SET search:abc123 <result> EX 300
   └── Cached with a 5-minute TTL

7. API returns the response to the user:
   └── {"results": [...], "source": "chromadb|cache", "cached": true|false}

Expected timings

The cache brings latency down from 50-200 ms to 1-5 ms for repeated queries. That's the reason Redis is in the stack.

Ports, volumes, and resources

Port map

Service     Internal port     Host port      External access
──────────────────────────────────────────────────────────
api         8000              8000           ✅ Always
chroma      8000              8100           ⚠️ Dev only
redis       6379              6379           ⚠️ Dev only

In production, only the API exposes a port to the host. ChromaDB and Redis are reachable only from the internal ai-network. You handle that with the override files (capsule 05).

Volume map

Volume          Service     Mount point           Content
──────────────────────────────────────────────────────────────
chroma-data     chroma      /chroma/chroma        Embeddings, collections
redis-data      redis       /data                 RDB snapshots, AOF

Named volumes guarantee the data survives between docker compose down and docker compose up. Without volumes, you'd lose every embedding and the whole cache each time you restart the stack.

Verifying persistence

# Add data
curl -X POST http://localhost:8000/documents \
  -H "Content-Type: application/json" \
  -d '{"documents": ["Docker containers are lightweight"], "ids": ["doc1"]}'

# Restart the stack
docker compose down
docker compose up -d

# Verify the data persisted
curl "http://localhost:8000/search?query=Docker&n_results=1"
# Should return doc1

If the data survives the restart, your volume is configured correctly.

Resource limits (production)

services:
  api:
    deploy:
      resources:
        limits:
          cpus: "1.0"
          memory: 512M
        reservations:
          cpus: "0.25"
          memory: 128M

  chroma:
    deploy:
      resources:
        limits:
          cpus: "1.0"
          memory: 1G
        reservations:
          cpus: "0.25"
          memory: 256M

  redis:
    deploy:
      resources:
        limits:
          cpus: "0.5"
          memory: 256M
        reservations:
          cpus: "0.1"
          memory: 64M

Resource limits prevent one service from consuming all the host's resources and hurting the others. ChromaDB gets more memory because vector search is memory-intensive.

Health checks and startup order

Health checks per service

Every service needs a health check that verifies it's genuinely functional:

Service     Type           Command/URL                         Interval
────────────────────────────────────────────────────────────────────────
redis       CMD            redis-cli ping                      10s
chroma      HTTP (curl)    curl http://localhost:8000/api/v1/heartbeat  15s
api         CMD (python)   python healthcheck.py               30s

The API has the longest interval because it depends on the other two services. There's no point checking the API every 10 seconds if its health check includes verifying Redis and ChromaDB.

Startup order with depends_on

services:
  redis:
    # Starts first — it has no dependencies

  chroma:
    # Starts second — it doesn't depend on Redis

  api:
    depends_on:
      redis:
        condition: service_healthy
      chroma:
        condition: service_healthy
    # Starts third — only once Redis AND ChromaDB are healthy

Startup sequence:
  t=0s    redis and chroma start in parallel
  t=10s   redis reports healthy (redis-cli ping → PONG)
  t=15s   chroma reports healthy (heartbeat → OK)
  t=16s   api starts (both dependencies healthy)
  t=46s   api reports healthy (healthcheck.py → OK)

Without condition: service_healthy, Docker only waits for the container to start — not for the service to be ready. The difference is crucial: a container can start in 1 second, but the application inside can take 10 seconds to become functional.

The project's file structure

Full view

production-ai-stack/
├── app/
│   ├── main.py              # FastAPI application (endpoints, logic)
│   └── healthcheck.py       # Health check script (doesn't require curl)
│
├── Dockerfile               # Multi-stage, non-root, health check
├── docker-compose.yml       # Base: 3 services, network, volumes
├── docker-compose.dev.yml   # Override: bind mounts, hot reload, debug ports
├── docker-compose.prod.yml  # Override: read-only, restart always, no debug ports
│
├── .env.example             # Documented environment variables
├── .dockerignore            # Files excluded from the build
├── README.md                # Operational documentation
│
└── verify-deployment.sh     # Checklist verification script

Every file has a clear purpose. There are no redundant or unnecessary files. This structure is what you'll implement in the following capsules.

Troubleshooting

"ChromaDB won't start: port already in use"

Both the API and ChromaDB use port 8000 internally. That's fine — they're different containers with different IPs. The conflict shows up if you map both to the same host port. The fix: map ChromaDB to host port 8100 (8100:8000).

"Redis connection refused from the API"

The API tries to connect to redis://redis:6379 but Redis isn't ready yet. Check that you have depends_on: { redis: { condition: service_healthy } } on the api service. Without condition: service_healthy, Docker only waits for the container to start — not for Redis to be accepting connections.

"ChromaDB: no such file or directory /chroma/chroma"

The chroma-data volume mounts at /chroma/chroma. If the directory doesn't exist inside the container, Docker creates it automatically. If the error persists, check that the mount point matches ChromaDB's configuration. The official image expects data in /chroma/chroma.

"The services start but api stays in 'waiting'"

If api never starts, it's because redis or chroma never reach the healthy state. Verify each health check individually:

docker compose up redis chroma -d
docker compose ps

If one of them shows (unhealthy), check its logs: docker compose logs redis or docker compose logs chroma.

Exercises

Exercise 1: Draw the architecture

Without looking at the capsule, draw on paper (or in text) the stack's architecture: the 3 services, which port each one uses, how they communicate, where the volumes are, and what the network is. Then compare it with the diagram in this capsule.

┌──────────────────────────────────────────┐
│              ai-network                   │
│                                           │
│  ┌──────┐     ┌────────┐    ┌───────┐   │
│  │ api  │────►│ chroma │    │ redis │   │
│  │ :8000│     │ :8000  │    │ :6379 │   │
│  │      │────────────────────►      │   │
│  └──────┘     └────────┘    └───────┘   │
│                                           │
│  Volumes: chroma-data, redis-data        │
└──────────────────────────────────────────┘

Host ports: api→8000, chroma→8100, redis→6379

Exercise 2: Trace a request

Describe step by step what happens when a user sends GET /search?query=python+basics&n_results=3. Include: which service receives the request, what query it makes to Redis, what happens on a cache hit vs a miss, what query it makes to ChromaDB, and what it returns to the user.

1. The request reaches the FastAPI API (port 8000)
   GET /search?query=python+basics&n_results=3

2. The API generates a cache key:
   key = "search:" + hash("python basics:3")
   key = "search:a7b3c9..."

3. The API queries Redis:
   GET search:a7b3c9...

4a. Cache HIT:
    Redis returns the cached result.
    The API returns to the user:
    {"results": [...], "source": "cache", "cached": true}
    Latency: ~2 ms

4b. Cache MISS:
    Redis returns nil.

5. (Only on a cache MISS) The API queries ChromaDB:
   collection.query(
     query_texts=["python basics"],
     n_results=3
   )
   ChromaDB returns the 3 most similar documents.

6. The API formats the results.

7. The API caches in Redis:
   SET search:a7b3c9... <result_json> EX 300
   (TTL = 5 minutes)

8. The API returns to the user:
   {"results": [...], "source": "chromadb", "cached": false}
   Latency: ~50-100 ms

Exercise 3: Identify the SPOF

SPOF = Single Point of Failure. If one of the stack's services dies, what happens to the others? Which is the most critical SPOF?

Exercise 4: Size the resources

Your server has 4 CPUs and 8 GB of RAM. Distribute the resource limits across the 3 services. Justify your decisions.

services:
  api:
    deploy:
      resources:
        limits:
          cpus: "1.5"
          memory: 2G
        reservations:
          cpus: "0.5"
          memory: 512M

  chroma:
    deploy:
      resources:
        limits:
          cpus: "1.5"
          memory: 4G
        reservations:
          cpus: "0.5"
          memory: 1G

  redis:
    deploy:
      resources:
        limits:
          cpus: "0.5"
          memory: 1G
        reservations:
          cpus: "0.25"
          memory: 256M

Exercise 5: Design the API's health check

The API's health check has to verify not just that FastAPI responds, but that Redis and ChromaDB are reachable. Design the /health endpoint so it returns a JSON with the status of each dependency.

@app.get("/health")
async def health_check():
    redis_ok = False
    chroma_ok = False

    try:
        r = redis.from_url(os.getenv("REDIS_URL", "redis://redis:6379"))
        redis_ok = r.ping()
    except Exception:
        redis_ok = False

    try:
        client = chromadb.HttpClient(
            host=os.getenv("CHROMA_HOST", "chroma"),
            port=int(os.getenv("CHROMA_PORT", "8000"))
        )
        client.heartbeat()
        chroma_ok = True
    except Exception:
        chroma_ok = False

    checks = {
        "api": True,
        "redis": redis_ok,
        "chromadb": chroma_ok,
    }
    all_healthy = all(checks.values())

    return JSONResponse(
        status_code=200 if all_healthy else 503,
        content={
            "status": "healthy" if all_healthy else "degraded",
            "checks": checks,
            "timestamp": datetime.now(timezone.utc).isoformat(),
        },
    )

{"status": "healthy", "checks": {"api": true, "redis": true, "chromadb": true}}

{"status": "degraded", "checks": {"api": true, "redis": false, "chromadb": true}}

Summary

• The Production AI Stack has 3 services: FastAPI API (custom), ChromaDB (official image), and Redis (Alpine image).
• The API is the only entry point. It receives requests, queries ChromaDB for semantic search, uses Redis as a cache, and returns responses.
• The data flow is: request → check the cache (Redis) → query the vector DB (ChromaDB) on a cache miss → cache the result → return the response.
• The services live on a custom network, ai-network, and communicate by service name (Docker Compose's automatic DNS).
• Named volumes (chroma-data, redis-data) guarantee persistence across restarts.
• The startup order uses depends_on with condition: service_healthy: Redis and ChromaDB first, the API after.
• Only the API has a custom Dockerfile. ChromaDB and Redis use official images.
• Resource limits prevent one service from consuming all the host's resources.
• In production, only the API exposes a port to the host. ChromaDB and Redis are reachable only from the internal network.

Additional resources

1. Docker Compose Networking — How networking works in Compose
2. ChromaDB Documentation — Official ChromaDB documentation
3. Redis Documentation — Official Redis documentation
4. Docker Compose depends_on — Controlling startup order
5. Docker Resource Constraints — Configuring resource limits
6. FastAPI Documentation — Official FastAPI reference
7. RAG Architecture Patterns — Retrieval-Augmented Generation patterns
8. Microservices Communication Patterns — Communication patterns between services