Module 7: Reliability Patterns & Production Checklist

8. Summary and Troubleshooting for Module 7

Description

In this module you built the reliability layer that separates an AI app that "works on my laptop" from one that "works in production with 500 concurrent users." The five patterns — error classification, retry, circuit breaker, rate limiting, and fallback — are not optional for real production. This capsule closes the module with the diagnosis of the most common problems you'll encounter, the AI-specific production checklist, and the connection to Module 8 where you'll integrate everything.

The 5 most common errors and how to diagnose them

Error 1: Retrying everything without discriminating

Symptom: Your app retries 401 and 400 errors, generating useless calls (and sometimes increasing the cost).

Diagnosis:

# In your logs, look for patterns like:
# llm_retry_attempt where error_type=AuthenticationError
# llm_retry_attempt where error_type=BadRequestError

# With jq on your JSONL logs:
jq 'select(.event == "llm_retry_attempt") | {error_type, attempt}' logs/app.json

Root cause: retry_if_exception_type((Exception,)) — catching too broadly.

Solution:

# ❌ Too broad:
retry=retry_if_exception_type(Exception)

# ✅ Only transient errors:
def is_retryable(e):
    from openai import APITimeoutError, RateLimitError, InternalServerError
    from src.infrastructure.llm_provider import LLMProviderError
    if isinstance(e, (APITimeoutError, RateLimitError, InternalServerError)):
        return True
    if isinstance(e, LLMProviderError):
        return e.should_retry
    return False  # Unknown → no retry by default

retry=retry_if_exception(is_retryable)

Error 2: Circuit breaker that doesn't trip when it should

Symptom: OpenAI has been down for 5 minutes, your app keeps sending requests (each one with 3 retries), the server is saturated, and the circuit never opens.

Diagnosis:

# The circuit never trips if:
# 1. The threshold is too high for the traffic volume
# 2. The failures don't match expected_exceptions
# 3. The circuit is per-request (new on each request, never accumulates)

# Verify that the circuit is a SINGLETON:
# Incorrect: create CircuitBreaker() inside the endpoint handler
# Correct: create CircuitBreaker() once at startup

# Check metrics:
circuit_breaker.get_metrics()
# → {"failure_count": 2, "failure_threshold": 5, "state": "closed"}
# If failure_count never goes up, the errors aren't being captured

Most common root cause: The CircuitBreaker is recreated on each request (for example, inside a function that's called per request).

Solution:

# ❌ Per-request CircuitBreaker (doesn't accumulate state):
@app.post("/analyze")
async def analyze(body: AnalyzeRequest):
    cb = CircuitBreaker(failure_threshold=5)  # ← NEW on each request
    with_cb = CircuitBreakerProvider(provider, cb)
    ...

# ✅ Singleton CircuitBreaker:
# In dependencies.py, outside any endpoint function:
_circuit_breaker = CircuitBreaker(name="openai", failure_threshold=5)

def get_llm_provider():
    return CircuitBreakerProvider(base_provider, _circuit_breaker)  # same cb

Error 3: Fallback that degrades silently and without logging

Symptom: You see in production that the quality of the responses has dropped. Nobody knows when it started. There are no fallback logs. The users don't see any indicator that the service is degraded.

Diagnosis:

# If there are no fallback logs, how do you know if it's being used?
# The answer is: you don't know.

# Look in your logs:
jq 'select(.event == "fallback_provider_used") | .fallback_used' logs/app.json | sort | uniq -c
# If there are no results, you have a visibility problem

# In the API responses, is there a "degraded" field?
# If all your responses have "degraded": false but the LLM is failing,
# you have a bug in the detection

Solution:

# 1. ALWAYS log when a fallback is used:
log.warning(
    "fallback_activated",
    primary_failed=self._names[0],
    fallback_used=name,
    request_id=get_request_id()  # From M5 — request tracing
)

# 2. Include degraded in the response:
return {
    **result,
    "degraded": provider.is_degraded,
    "degraded_reason": "Using backup model" if provider.is_degraded else None
}

# 3. Fallback metrics in the health check:
# /health/deps → show fallback_activation_count

Error 4: Cosmetic health check

Symptom: Kubernetes shows all pods as "ready", but the /analyze endpoint fails with 500 because OpenAI is down.

Diagnosis:

# If your health check is:
# GET /health/ready → {"status": "ok"}  # Always 200

# And your real endpoint is:
# POST /analyze → 500 (because OpenAI fails)

# Kubernetes thinks the pod is ready and keeps sending traffic
# The result: all requests fail until K8s detects the problem

Solution:

# /health/ready must check dependencies:
@router.get("/ready")
async def readiness():
    checks = {}
    
    # Check circuit breakers
    cb_metrics = get_circuit_breaker_metrics()
    any_open = any(m["state"] == "open" for m in cb_metrics.values())
    
    if any_open:
        return JSONResponse(
            {"status": "not_ready", "reason": "circuit_open"},
            status_code=503  # K8s won't send traffic
        )
    
    return {"status": "ready"}

# With this:
# - If circuit is open → 503 → K8s stops sending traffic to the pod
# - Traffic goes to other pods (or the load balancer's fallback triggers)

Error 5: Rate limiting that blocks instead of throttling

Symptom: Under load, your app "freezes" — requests take 30 seconds to respond (the rate limiter's max_wait_seconds), and many of them time out before being processed.

Diagnosis:

# If max_wait_seconds=30 and rate=1 req/s,
# and 60 simultaneous requests arrive:
# - The first ones pass immediately (burst)
# - The rest wait in the queue: up to 60 seconds
# - Your API timeout may be 30s → the last 30 requests time out

# Check the bucket utilization:
bucket.get_metrics()
# → {"utilization_percent": 100, "total_rejected": 0, "total_waited_seconds": 450}
# If waited_seconds is high, the requests are waiting too long

Solution:

# Option 1: Reduce max_wait and reject fast (for interactive APIs)
rate_limited = RateLimitedProvider(
    inner=provider,
    requests_per_minute=60,
    max_wait_seconds=5.0  # ← Reject if it can't process in 5s
)

# Option 2: Add the Retry-After header in the rejected response
# So the client knows when to retry

# Option 3: Make the queue visible to the user
# "Your request is in the queue. Position: 15/30."

AI Production Checklist: what separates dev from prod

Minimum checklist (before any launch)

RELIABILITY
├── [ ] Retry implemented with tenacity
│   ├── [ ] stop_after_attempt(3-5) — no infinite retry
│   ├── [ ] wait_random_exponential(min=1, max=30) — backoff + jitter
│   ├── [ ] retry only transient errors (Timeout, RateLimit, 5xx)
│   └── [ ] before_sleep callback with logging of each retry
│
├── [ ] Circuit breaker implemented
│   ├── [ ] failure_threshold configured (5-10 typical)
│   ├── [ ] recovery_timeout configured (30-60s)
│   ├── [ ] The CircuitBreaker is a singleton (not per-request)
│   └── [ ] Logs when it opens and closes
│
├── [ ] Client-side rate limiting
│   ├── [ ] Below 80% of the API limit
│   ├── [ ] max_wait_seconds reasonable for the use case
│   └── [ ] Per model if multiple models are used
│
├── [ ] Fallback chain
│   ├── [ ] At least primary + secondary provider
│   ├── [ ] Static fallback as a last resort
│   ├── [ ] Logs each fallback activation
│   └── [ ] Communicates degradation to the user (degraded field in response)
│
OBSERVABILITY (from M5)
├── [ ] Structured logging (JSON) configured
├── [ ] request_id in all logs
├── [ ] Cost logged per request
├── [ ] Cost alerts configured
│
GUARDRAILS (from M4)
├── [ ] Input validation on all endpoints
├── [ ] Output validation with Pydantic
├── [ ] Content policy check on endpoints with user input
│
CLEAN ARCHITECTURE (from M6)
├── [ ] Domain doesn't depend on concrete providers
├── [ ] Config in pydantic-settings with validation
├── [ ] Domain tests with MockProvider (no real calls)
│
HEALTH CHECKS
├── [ ] /health/live — returns 200 whenever the process is alive
├── [ ] /health/ready — returns 503 if circuit is open or deps are down
└── [ ] /health/deps — checks OpenAI, rate limits, budget

Production operations checklist (ongoing)

MONITORING
├── [ ] Dashboard with: latency p50/p95, error rate, cost/day, fallback rate
├── [ ] Alert if error rate > 5% for 5 minutes
├── [ ] Alert if cost/day > threshold (e.g. $50/day)
├── [ ] Alert if the circuit breaker opens
│
INCIDENT RESPONSE
├── [ ] Documented runbook: what to do if OpenAI is down?
├── [ ] Runbook: what to do if the budget is exceeded?
├── [ ] Alerts channel configured (Slack, PagerDuty, etc.)
│
MAINTENANCE
├── [ ] Rate limits reviewed monthly (OpenAI changes them)
├── [ ] Circuit breaker thresholds adjusted based on real traffic
└── [ ] Costs reviewed weekly

Diagnostic tree: what's failing?

Symptom: My app is returning many 500 errors
│
├── Do the logs show llm_retry_attempt?
│   ├── YES → Retry is active. Are the retries exhausted?
│   │         ├── YES → Is it an outage? → Is the circuit open?
│   │         │         ├── YES (circuit open) → Normal. Wait for recovery.
│   │         │         └── NO → Increase failure_threshold or investigate
│   │         └── NO → Retry works. Is the problem something else?
│   │
│   └── NO → Retry isn't active. Is it configured?
│             └── Verify that RetryProvider wraps the OpenAIProvider
│
├── Do the logs show circuit_breaker_opened?
│   ├── YES → The circuit opened. Is there a fallback?
│   │         ├── YES → Does the fallback work? See fallback_provider_used in logs
│   │         └── NO → Implement fallback (capsule 06)
│   └── NO → The circuit doesn't trip. Is it a singleton?
│             └── Verify that CircuitBreaker is created once, not per request
│
└── Do the logs show client_rate_limit_rejected?
    ├── YES → The rate limiter is rejecting. Is max_wait too short?
    │         └── Increase max_wait or raise the rate limit
    └── NO → The problem isn't rate limiting

Module summary

#	Capsule	What you learned
01	Introduction	The 5 patterns, composition, how M6's DI facilitates everything
02	Error Handling	Taxonomy: TRANSIENT, INPUT_ERROR, AUTH_ERROR, OUTPUT_ERROR
03	Retry + Backoff	tenacity, exponential + jitter, only transients, retry logging
04	Circuit Breaker	States CLOSED→OPEN→HALF_OPEN, singletons, composition with retry
05	Rate Limiting	Token bucket, 80% safety margin, queue vs reject, budget control
06	Fallbacks + Health	Fallback hierarchy, communicate degradation, liveness vs readiness
07	Project	Complete assembly in dependencies.py, composition tests
08	Troubleshooting	The 5 most common errors, AI production checklist

What's coming: Module 8 (Integrative Project)

Module 8 is the culmination of the entire guide. In it you'll integrate:

Tests (M2-M3): unit tests, integration tests, the MockProvider as a base
Guardrails (M4): input/output validation on all endpoints
Logging + Observability (M5): structured logging, request tracing, cost tracking
Clean Architecture (M6): separation into layers, DI, config with pydantic-settings
Reliability (M7): the complete layer of retry/circuit/rate limit/fallback

The result is a complete production-ready AI system that you can use as a template for real projects or as a portfolio piece that demonstrates you understand all the aspects of bringing AI to production.

Exercises

Exercise 1: Quick diagnosis

You're in production and you see these logs. What's the diagnosis and the immediate action?

{"event": "llm_retry_attempt", "attempt_number": 3, "exception_type": "RateLimitError"}
{"event": "llm_retry_attempt", "attempt_number": 3, "exception_type": "RateLimitError"}
{"event": "circuit_breaker_opened", "circuit": "primary", "failure_count": 5}

See solution

Diagnosis: The primary provider is being rate-limited persistently. The retries don't solve the problem because the rate limit doesn't free up between attempts. After 5 failures, the circuit opens.

Immediate action:

Check whether there's a traffic spike (why so many requests?)
Check the OpenAI dashboard — are you close to the limit?
If the fallback is active, the service keeps working with degradation
Reduce the client-side rate limit temporarily or increase the tier in OpenAI

Exercise 2: Production checklist

Before deploying your reliability layer, verify each item. Check the ones you have:

- [ ] Retry: max_attempts ≤ 5 (more = unacceptable latency)
- [ ] Retry: min_wait ≥ 1s (less = thundering herd)
- [ ] Circuit breaker: failure_threshold ≥ 5 (less = false positives)
- [ ] Circuit breaker: is a singleton (not recreated per request)
- [ ] Rate limiter: uses 80% of the API's real limit
- [ ] Fallback: has at least one level after the primary
- [ ] Health: /health/live does NOT check external dependencies
- [ ] Health: /health/ready DOES check external dependencies
- [ ] Logs: each retry/circuit/rate event is logged with request_id
- [ ] Tests: the complete chain is tested with fast mocks

See verification guide

If any is missing:

Retry without min_wait: add min_wait_seconds=1.0 in RetryProvider
Circuit not a singleton: move to a global or module-level variable in dependencies.py
Rate limiter at 100%: change to requests_per_minute = int(api_limit * 0.8)
Liveness checks OpenAI: simplify /health/live to just {"status": "alive"}

Additional resources

tenacity Documentation — Complete retry library
Circuit Breaker (Martin Fowler) — The canonical article
Release It! (Michael Nygard) — The book on reliability in distributed systems
OpenAI Rate Limits Guide — Current limits and handling strategies
Kubernetes Probes — Liveness and readiness in K8s
AWS: Exponential Backoff and Jitter — The canonical article on jitter

← Previous7. Project: Reliability Layer Next →1. Production Checklist