Module 5: Reliability at Scale

Project: Reliability Design Document

Next →

Overview

Up to here you've seen the 6 pillars of reliability for AI: load balancing with deep health checks, queue-based processing, technical and budget rate limiting, circuit breakers that detect degradation, retry with idempotency, and graceful degradation. Each one separately.

This project integrates them. You're going to produce a complete Reliability Design Document applied to a concrete case. It's the deliverable a Tech Lead would present at a design review, or that your future self would consult when something goes down in production.

It's not code — it's design. But the design must be concrete enough that a team can implement it with confidence. Diagrams, numbers, justified decisions, not empty abstractions.

By the end of the project you'll have:

  • A complete Reliability Design Document in Markdown (5-10 pages)
  • Diagrams showing the request flow and the reliability components
  • A table of applied patterns with case-by-case justification
  • A basic runbook for common incidents

The case to design

System: an AI-Powered Customer Support for a B2B SaaS.

  • Users: 200 customer support agents across 5 tenant companies
  • Function: agents write queries, the RAG system over the client's internal documentation returns a suggested response
  • Traffic: 600 queries/hour peak (10/min), 100/hour average
  • SLA: P95 < 8s, availability 99.5% monthly (allows ~3.5h/month of accumulated downtime)
  • Current stack:
    • API: FastAPI on 3 EC2 instances with a load balancer
    • LLM: OpenAI gpt-4o-mini (primary provider)
    • Vector DB: Pinecone (managed)
    • Cache: Redis Cluster
  • Constraints:
    • Budget: $1,500/month on OpenAI tokens
    • The enterprise client (one of the 5) demands that no downtime exceed 30 consecutive minutes

Document template

Your deliverable is a Markdown with this structure. Fill each section with specific decisions for the case above.

1. Executive summary (½ page)

# Reliability Design — AI Customer Support Assistant

## Summary

This document describes the reliability strategy of the AI Customer Support
Assistant. It covers the 6 pillars: load balancing, queue-based processing,
rate limiting, circuit breakers, retry and idempotency, and graceful degradation.

The system is designed to meet an SLA of P95 <8s and 99.5% monthly
availability, within a budget of $1,500/month on LLM tokens.

The decisions prioritize: (1) a consistent experience for support agents
(tolerable latency, no visible failures), (2) budget protection
(avoid bursts that drain the budget), (3) automatic recovery (less
human intervention).

2. Architecture overview (1 page)

A diagram showing the main components and the flow of a request.

                    Agent (browser)
                          │
                          ▼
                    Load Balancer (L7, AWS ALB)
                  ┌───────┼───────┐
                  ▼       ▼       ▼
              API-1    API-2    API-3   (FastAPI, 3 instances)
                  │       │       │
                  └───────┼───────┘
                          ▼
                    Redis Queue
                          │
                          ▼
                    Workers (4 instances)
                          │
                          ├─→ Pinecone (vector search)
                          ├─→ OpenAI (LLM, with circuit breaker)
                          └─→ Anthropic (fallback with circuit breaker)
                          │
                          ▼
                    Result store (Redis)
                          │
                          ▼
                    Agent (poll or WebSocket)

For each component, a one-line purpose.

3. Load balancing (½ page)

Decisions:

  • Type: Layer 7 (AWS Application Load Balancer)
  • Strategy: least-connections (variable latencies 2-30s across queries)
  • Health check: deep (/ready) with verification of OpenAI + Pinecone + Redis, cached 30s
  • Sticky sessions: NO (stateless API)
  • Drain time: 60s (kubernetes terminationGracePeriodSeconds)

4. Queue-based processing (1 page)

Flow design:

StepComponentTarget latency
Agent sends a queryBrowser → ALB → API<100ms
API validates + enqueuesAPI → Redis<200ms
API responds 202 with job_idAPI → Agent<300ms total
Worker dequeuesRedis → Workerimmediate
Worker does RAG + LLMWorker → Pinecone + OpenAI3-25s
Worker stores resultWorker → Redis result store<100ms
Agent receives resultPolling every 1s or WebSocket<1s after the previous step

Worker calculation:

  • Peak: 10 queries/min = 0.167 q/s
  • Average duration: 12s
  • Utilization target: 70%
  • Workers needed: (0.167 × 12) / 0.7 ≈ 2.9 → round up to 4

Notification to the agent: polling (simpler for this case, scales well with 200 agents).

5. Rate limiting (1 page)

Multi-level:

LevelLimitAlgorithmJustification
Per agent (user)30 queries/hourToken bucketFairness between agents, prevents one agent from saturating
Per tenant (company)200 queries/hourToken bucketFairness between clients, scales with the plan
Per endpoint /search5 q/s globalToken bucketProtects general capacity
OpenAI (vendor)600 RPM (of the 10K available)Centralized token bucketMargin for spikes, respects the OpenAI tier
OpenAI budget$1500/month = $0.0006/s sustainedCents bucketHard budget cap

Behavior when the limit is reached:

  • Per agent / tenant: hard reject 429 with Retry-After: 60
  • Per global endpoint: queue (defer, don't reject)
  • Budget: if <10% remaining in the month, degrade to Cache-First mode before hard reject

6. Circuit breakers (1 page)

Three configured circuits:

CircuitDownstreamThresholdsCooldown
openai_primaryOpenAI gpt-4o-minierror_rate >50%, P95 >15s, 30 req minimum60s
anthropic_fallbackClaude 3 Haikuerror_rate >50%, P95 >20s90s
pinecone_searchPineconeerror_rate >30%, P95 >3s30s

Fallback cascade when a circuit is open:

openai_primary OPEN
   ↓
try anthropic_fallback
   ↓ (also open)
try cache (Redis)
   ↓ (cache miss)
canned response by category
   ↓ (no category matches)
degraded message

Distributed state in Redis: all instances share the circuit's state.

7. Retry and idempotency (½ page)

Retry per layer:

LayerMax retriesBackoff baseJitter
OpenAI call32s exponentialFull jitter
Anthropic call32s exponentialFull jitter
Pinecone2200ms exponentialFull jitter
Redis2100msFull jitter

Idempotency:

  • Each job has a job_id UUID
  • The API checks the client's optional idempotency_key; maps it to a job_id
  • The worker checks the job's status before processing (skip if already processed)
  • DLQ after 5 total retries per job

DLQ:

  • A separate queue dlq:queries
  • Alert to Slack #ops when it grows
  • Manual processing with a replay script

8. Graceful degradation (1 page)

4 levels:

LevelTriggerBehaviorUI message
N0 Normal—Normal flow(nothing)
N1 SlowOpenAI P95 >6s for 1minTimeout up to 45s, UI shows "processing...""Processing your query..."
N2 Cache-firstCircuit openai_primary open OR queue >30 sustainedCache hit if it exists → Anthropic → canned"Fast response" badge
N3 DegradedBoth LLM circuits open OR budget <5% remainingCanned response by detected category"Limited operation, try again in more detail or later"
N4 Service downPinecone down OR Redis downOnly static info, queue everything for deferred processing"Technical problems, status: [link]"

Transitions:

  • Going up: 30s sustained under the level's conditions
  • Coming down: 5 min sustained without the level's conditions
  • Manual override available for operators

9. Monitoring and alerts (½ page)

Mandatory metrics:

  • Queue depth (alert if >50 sustained for 2min)
  • DLQ size (alert if >0)
  • Circuit state (alert if any circuit is open >5min)
  • LLM P95 latency (alert if >10s sustained)
  • LLM error rate (alert if >10% sustained)
  • Budget consumption rate (alert if the projection is >100% of the monthly)
  • Degradation level (alert when it goes up from N0→N1+)

10. Basic runbook (1 page)

SymptomInvestigationAction
High queue depthCheck workers status, down?Restart workers, scale up
openai_primary openCheck the OpenAI status pageVerify the fallback works; if OpenAI is prolonged, consider lowering the budget threshold
DLQ >0Inspect the jobs in the DLQCategorize errors, replay the recoverable ones manually
Budget >80% spent by mid-monthCheck if there's abuse from a tenantNotify the tenant, consider a stricter throttle
System enters N3 for no clear reasonDegradation manager logsVerify the circuits aren't "stuck" open

11. Appendices

  • A: Redis configuration (ports, cluster topology)
  • B: Rate limiting Lua scripts (the ones from the exercise in M5-04)
  • C: pybreaker configuration / circuit state schema in Redis
  • D: List of canned responses by category (initial knowledge base)

How to work on this project

I suggest this order:

  1. Skim the template (5 min): familiarize yourself with the sections.
  2. Sketch the diagram (15 min): draw the main flow by hand or with tldraw.
  3. Decisions per section (1.5-2 hrs): complete each section with concrete numbers for the case.
  4. Cross-consistency (15 min): check that the degradation levels are consistent with the defined triggers, that the rate limits respect the budget, that the circuits and queues are aligned.
  5. Runbook (15 min): think "if this goes down at 3am, what do I do?"

Total: ~3 hours of work.

Evaluation criteria

To ensure your document is production-ready, self-assess:

  • The architecture diagram is legible and shows all the reliability components
  • Each decision has a justification (not just "I use queues" but "queues because we have 5-30s of latency and a 3s UI budget")
  • There are concrete numbers (how many workers, how many RPS, what thresholds) — not generic
  • It covers the 6 pillars of M5 (load balancing, queues, rate limiting, circuits, retry, degradation)
  • The runbook describes responses to at least 5 different scenarios
  • Metrics and alerts are automatable (what to measure, what threshold, what to notify)
  • A new team member could implement the system following this document

Example of a complete page: section 6 (Circuit Breakers)

To calibrate the level of detail, this is what a well-done section looks like:

## 6. Circuit Breakers

### 6.1 Configured circuits

Three independent circuits, one for each critical downstream:

| Circuit | Downstream | Tracked metrics | Thresholds | Cooldown |
|---------|-----------|------------------|----------|----------|
| `openai_primary` | OpenAI gpt-4o-mini | error rate, P95 latency, timeout rate, empty responses | error >50% OR P95 >15s OR timeouts >30%, min 30 reqs | 60s + jitter ±15s |
| `anthropic_fallback` | Claude 3 Haiku | error rate, P95 latency | error >50% OR P95 >20s, min 20 reqs | 90s + jitter ±15s |
| `pinecone_search` | Pinecone vector search | error rate, P95 latency | error >30% OR P95 >3s, min 50 reqs | 30s + jitter ±5s |

### 6.2 Distributed state

Each circuit's state is stored in Redis (`circuit:{name}:state` and
`circuit:{name}:events`). All worker instances query the same
state, avoiding each worker maintaining its own independent view.

### 6.3 Fallback cascade

When a request requires the LLM, the order of attempts is:

1. **openai_primary** — whenever the circuit is CLOSED
2. **anthropic_fallback** — if openai_primary is OPEN or fails
3. **Redis cache** (`cache:response:{hash(prompt)}`) — if both LLM circuits are open
4. **Canned response** by category — classifies the prompt with keywords, returns a template
5. **Degraded message** — last option

This cascade is implemented in `worker.py`, function `process_query()`.

### 6.4 Monitoring

Metrics exported to Prometheus:
- `circuit_state{name="X"}` — gauge (0=closed, 1=half-open, 2=open)
- `circuit_transitions_total{name="X", from="Y", to="Z"}` — counter
- `circuit_rejections_total{name="X"}` — counter (how many requests rejected by the circuit)
- `circuit_fallback_usage_total{from="X", to="Y"}` — counter

Alerts:
- Severity HIGH: any circuit OPEN >5min
- Severity MEDIUM: a circuit with >3 openings in the last hour
- Severity LOW: a circuit uses the fallback >20% of the time

That level of detail per section is what we expect.

Evidence of success by the end of M5

You'll know you finished well if you can:

  • ✅ Deliver the complete document (all sections filled in)
  • ✅ Defend every number you used (e.g., "why 4 workers?" → show the calculation)
  • ✅ Explain the complete fallback cascade
  • ✅ Show where the single points of failure are (ideally: none critical without mitigation)
  • ✅ Resolve at least 3 runbook scenarios without re-thinking the architecture

Connection with the rest of the guide

This Reliability Design Document isn't discarded at the end of M5. It's the reliability section of the Capstone Architecture Design (M8). When you reach M8, you take this document, adapt it to the Knowledge Assistant system of the Capstone (Slack/Discord, Agentic RAG), and integrate it with the scaling strategy (M3), integration patterns (M4), trade-off decisions (M7).

Work done well here saves work later on.

Next module

Module 6 — Real-World Integrations with Slack/Discord. Reliability designed in the abstract. Now we land it on concrete production channels: Slack and Discord. Each one has specific constraints (timeouts, rate limits, OAuth, message format) that require adapting your generic patterns.

Resources

  1. Architecture Decision Records template — for documenting decisions.
  2. Diagrams as Code — Python lib to generate reproducible diagrams.
  3. tldraw — quick sketch of diagrams if you prefer by hand.
  4. Google SRE Workbook — for runbooks and postmortems.
  5. Incident Response Cheat Sheet — professional runbook format.
← PreviousGraceful degradation by levelsNext →Introduction to Real-World Integrations — Slack/Discord