1. Introduction: Reliability Patterns

Description

"It works on my machine" doesn't mean "it works in production." In production, things fail — it's not a possibility, it's a certainty. The question isn't whether OpenAI will return a 429 error, but when. Reliability patterns don't prevent those failures: they let you handle them so your service keeps working or fails in a controlled way. In this module you'll build that resilience layer on top of the clean architecture you implemented in Module 6.

By the end of this module, you'll have:

• A system that retries transient errors without saturating the API
• A circuit breaker that detects outages and stops sending useless requests
• Client-side rate limiting to control your spending and traffic
• A fallback chain that keeps the service alive even when the primary provider goes down
• Real health checks that Kubernetes can use to decide whether your pod is ready

The real failure modes of LLM APIs

Before writing code, you need to understand the scenarios you'll face. These aren't hypothetical — they're situations that happen regularly in production:

Scenario 1: Rate limit under pressure
  - Your app has 100 simultaneous active users
  - They all generate requests at the same time (Monday 9am)
  - OpenAI returns 429 (Too Many Requests)
  - Without retry: everyone sees error 500
  - With retry + backoff: requests spread out, most complete

Scenario 2: Timeout on a long prompt
  - A user sends a 50-page document
  - The model takes 45s to process it
  - Your client has a 30s timeout
  - Without timeout handling: uncaught exception, error 500
  - With timeout retry: retries with a truncated prompt

Scenario 3: 2-hour outage
  - OpenAI has an infrastructure incident
  - Without circuit breaker: your app sends 5000 failed requests in 2 hours
    (each request does 3 retries = 15000 useless calls to OpenAI)
    (cost: your server processing requests that will never work)
  - With circuit breaker: after 5 consecutive failures, the circuit opens
    The next calls fail immediately (without calling OpenAI)
    After 60s, the circuit tries a call → if it works, it closes

Scenario 4: Traffic spike
  - Your blog post goes viral, 1000 users in 10 minutes
  - Without rate limiting: your 1000 simultaneous requests all fail with 429
  - With rate limiter: requests are throttled, 8/second are processed
    Users wait a bit, but get a response

Scenario 5: Malformed response
  - The LLM returns text instead of JSON (for some reason)
  - Without fallback: parse exception, error 500
  - With fallback: tries to parse, if it fails returns a default response
    And logs it to investigate

The 5 patterns of the module

┌──────────────────────────────────────────────────────────────┐
│                      RELIABILITY LAYER                       │
│                                                              │
│  ┌─────────────┐  ┌──────────────┐  ┌────────────────────┐   │
│  │   RETRY     │  │   CIRCUIT    │  │   RATE LIMITING    │   │
│  │  + BACKOFF  │  │   BREAKER    │  │  (Token Bucket)    │   │
│  │             │  │              │  │                    │   │
│  │  Transient  │  │  Persistent  │  │   Throttling       │   │
│  │ → retries   │  │  → fails     │  │   → controls       │   │
│  │   with wait │  │    fast      │  │     speed          │   │
│  └─────────────┘  └──────────────┘  └────────────────────┘   │
│                                                              │
│  ┌────────────────────────────────────────────────────────┐  │
│  │                     FALLBACK CHAIN                     │  │
│  │                                                        │  │
│  │  Primary → Secondary Model → Cached Response → Static  │  │
│  └────────────────────────────────────────────────────────┘  │
│                                                              │
│  ┌────────────────────────────────────────────────────────┐  │
│  │                     HEALTH CHECKS                      │  │
│  │  Readiness + Liveness + real Dependency checks         │  │
│  └────────────────────────────────────────────────────────┘  │
└──────────────────────────────────────────────────────────────┘

Why M6's clean architecture makes this easy

# BEFORE M6 (without clean architecture):
# If you wanted to add retry to your app, you'd have to modify:
# - The god function in main.py (where the LLM call is mixed with everything)
# - Every place where the LLM is called
# - The tests (which depend on the internal implementation)

# AFTER M6 (with clean architecture and DI):
# To add reliability, you just create LLMProvider wrappers:

from src.infrastructure.llm_provider import LLMProvider

class RetryProvider:
    """Wraps any LLMProvider with retry logic."""
    def __init__(self, inner: LLMProvider, max_attempts: int = 3):
        self._inner = inner
        self._max = max_attempts
    
    def complete(self, messages: list[dict], **kwargs) -> str:
        # Adds retry without modifying domain or processing
        ...

class CircuitBreakerProvider:
    """Wraps any LLMProvider with a circuit breaker."""
    def __init__(self, inner: LLMProvider, failure_threshold: int = 5):
        self._inner = inner
        ...
    
    def complete(self, messages: list[dict], **kwargs) -> str:
        ...

# In dependencies.py (the only place that changes):
def get_llm_provider() -> LLMProvider:
    base = OpenAIProvider.from_settings(settings)
    with_retry = RetryProvider(base, max_attempts=3)
    with_circuit_breaker = CircuitBreakerProvider(with_retry, failure_threshold=5)
    return with_circuit_breaker

# The domain (sentiment_service.py) doesn't change.
# The domain tests don't change.
# Only dependencies.py changes.

When to use each pattern

Is the error transient (lasts seconds)?
└─ YES → RETRY with backoff
   Examples: momentary timeout, rate limit that frees up soon

Has the service failed many times in a row?
└─ YES → CIRCUIT BREAKER
   Examples: outage, server down, prolonged rate limit

Can traffic exceed the API limits?
└─ YES → RATE LIMITING
   Examples: traffic spikes, sustained high load

Does the primary provider fail and you need to keep serving?
└─ YES → FALLBACK
   Examples: OpenAI outage, model unavailable

Do you need to know if the system can receive traffic?
└─ YES → HEALTH CHECKS
   Examples: Kubernetes readiness, monitoring, alerts

Composition: the patterns work together

# The correct composition sequence:

# 1. Rate Limiter: do I have capacity for this request?
#    If NO → reject with 503, don't spend resources
#    If YES → continue

# 2. Circuit Breaker: is the service available?
#    If OPEN → go to fallback immediately
#    If CLOSED/HALF-OPEN → continue

# 3. Retry: did the request fail due to a transient error?
#    If YES → retry with backoff
#    If NO (permanent error) → propagate the error

# 4. Fallback: did all retries fail?
#    PRIMARY → SECONDARY → CACHED → STATIC DEFAULT

# In code:
def get_llm_provider() -> LLMProvider:
    base = OpenAIProvider.from_settings(settings)  # The real call
    
    # Apply in the correct order (from outside to inside):
    with_retry = RetryProvider(base, max_attempts=3)           # Innermost
    with_cb = CircuitBreakerProvider(with_retry, threshold=5)  # Wraps retry
    with_fallback = FallbackProvider(with_cb, fallback_model=secondary)  # Outermost
    
    return with_fallback

# Flow of a request:
# FallbackProvider.complete()
#   → CircuitBreakerProvider.complete()
#     → [Circuit CLOSED: passes]
#     → RetryProvider.complete()
#       → Attempt 1: TimeoutError → wait 1s
#       → Attempt 2: TimeoutError → wait 2s  
#       → Attempt 3: success → return
#   → [After 5 circuit failures, it opens]
#   → FallbackProvider detects CircuitOpenError
#   → Tries secondary_provider
#   → Secondary works → return with degraded=True flag

Module prerequisites

# Install the reliability dependencies
pip install tenacity     # Retry with backoff
pip install pybreaker    # Circuit breaker (optional, there's also a custom implementation)

Module roadmap

Exercises

Exercise 1: Classify the failure modes

For each scenario, indicate the correct pattern:

1. OpenAI returns 429 after a traffic spike
2. The app has been receiving 500 errors from OpenAI for 10 minutes
3. 200 users make simultaneous requests to an endpoint with a 50/min limit
4. The LLM returns malformed JSON in 2% of requests

Exercise 2: Design the composition

For an app with these characteristics:

• OpenAI limit: 100 RPM
• OpenAI is expected to have ~99.5% uptime (outage ~4h/month)
• The LLM has ~1% malformed responses
• You have a cheaper backup model

Design the provider composition chain:

# From inner to outer:
primary = OpenAIProvider(model="gpt-4o", ...)
with_retry = RetryProvider(primary, max_attempts=3)  # For transient errors
with_cb = CircuitBreakerProvider(with_retry, failure_threshold=5, recovery_timeout=60)
                                            # For the 4h outages
rate_limiter = RateLimitedProvider(with_cb, rpm=80)  # 80% of the 100 RPM limit
fallback = FallbackProvider(
    rate_limiter,
    secondary=OpenAIProvider(model="gpt-4o-mini", ...),  # Cheaper backup
    static_default={"sentiment": "unknown", "score": 0.0, "confidence": 0.0}
)

Exercise 3: Identify the wrong pattern

In each case, someone chose the wrong pattern. Explain why it's wrong and which one you would use:

1. A dev set retry with 10 attempts for an AuthenticationError (401)
2. A dev set a circuit breaker with failure_threshold=1 for a sentiment analysis LLM
3. A dev removed the rate limiter because "OpenAI already limits me"

Exercise 4: Calculate the impact of an outage

Your app has 50 users/minute. OpenAI has a 15-minute outage. Calculate for each scenario:

Troubleshooting

"I don't know if my error is transient or permanent"

Apply this rule: if you send the same request 5 minutes later and it could work → it's transient. If you send the same request 100 times and it always fails → it's permanent. Timeouts and 429s are transient; 401s and 400s are permanent.

"Do I need to implement all the patterns?"

Not necessarily all from day 1. The minimum priority for production is:

1. Error classification (always — without it you can't decide anything)
2. Retry with backoff (almost always — protects against transients)
3. Health checks (always — Kubernetes needs them)
4. Circuit breaker (when you expect frequent outages)
5. Rate limiting + Fallback (when you have high traffic or backup models)

"How do I test all this if I can't force OpenAI errors?"

Use mocks. The MockProvider from M6 lets you simulate any error:

from src.infrastructure.llm_provider import LLMProviderError
from src.infrastructure.error_classifier import ErrorCategory

def create_failing_provider(error_type: ErrorCategory):
    """Create a provider that fails with the error type you need to test."""
    mock = MockProvider()
    original = mock.complete
    def failing(messages, **kwargs):
        raise LLMProviderError(
            "Simulated error",
            category=error_type,
            should_retry=(error_type == ErrorCategory.TRANSIENT)
        )
    mock.complete = failing
    return mock

"Does the composition order matter?"

Yes, it matters a lot. The correct order is: FallbackProvider(RateLimiter(CircuitBreaker(Retry(Base)))). If you invert the order of circuit breaker and retry, the retry cancels out the circuit's benefit — it keeps retrying even though the service is down. Capsule 04 explains this in detail.

Summary

• Failure is not if, but when: the patterns don't prevent failures, they handle them
• 5 key patterns: retry (transients), circuit breaker (persistent), rate limiting (throttling), fallback (controlled degradation), health checks (observability)
• M6's DI is the key: wrap providers without touching domain or processing
• Composition: the patterns stack, each handling its own failure type

Additional resources

1. Release It! (Michael Nygard) — The reference book for reliability patterns
2. tenacity Documentation — The retry library
3. Circuit Breaker Pattern (Martin Fowler) — The pattern explained
4. OpenAI Rate Limits — The real limits
5. Designing Data-Intensive Applications (Kleppmann) — Reliability in distributed systems