Circuit breakers for LLM providers

Overview

Your queue works, your rate limiter protects. But there's a scenario neither of them solves: the LLM provider is responding, just badly. Latency of 30s instead of 2s. An error rate of 30% instead of <1%. Your system accepts requests, passes them to the worker, the worker waits 30s, eventually receives an error or a degraded response. Your queue fills up. Your users wait minutes. Everything goes down slowly.

A circuit breaker cuts that chain. It detects that the provider is degraded, stops sending it requests for a while, and lets your system use a fallback (another provider, cache, degraded response). After a cooldown, it tries again. If it works, it goes back to normal flow. If not, it stays open.

The pattern is standard. The critical thing for AI: traditional circuit breakers only count HTTP errors. For LLM providers, you also need to trip on excessive latency — because a degraded provider can respond 200 OK but take 30s.

By the end you'll be able to:

• Implement the classic 3-state pattern (closed, open, half-open)
• Configure the right thresholds for AI (errors + latency + timeouts)
• Combine circuit breakers with a fallback (second provider, cache, degradation)
• Distribute the circuit breaker's state across multiple instances

---

The classic pattern: 3 states

       requests OK
   ┌──────────────────────────────┐
   │                              │
   ▼                              │
┌────────┐  errors >threshold ┌─────────┐  timeout cooldown  ┌─────────┐
│ CLOSED │ ─────────────────▶│  OPEN   │ ─────────────────▶ │HALF-OPEN│
│        │                   │         │                     │         │
└────────┘ ◀─────────────────│         │ ◀─────────────────  │         │
   ▲       error in half-open │         │   success in test  │         │
   │                          └─────────┘                     └─────────┘
   │                                                              │
   └──────────────────────────────────────────────────────────────┘
                          requests OK

Closed (normal)

Traffic flows normally. You count errors in a time window. If they exceed a threshold, you transition to OPEN.

Open (provider considered down)

You skip the provider. Every request immediately gets a fallback response (another provider, cache, error). You wait N seconds (cooldown). After that, you transition to HALF-OPEN.

Half-Open (testing recovery)

You let some requests through to the provider. If they succeed → you go back to CLOSED. If any fails → you go back to OPEN with the cooldown reset.

---

The metrics that trip the circuit (what changes in AI)

Generic circuit breakers count HTTP errors: 4xx (client errors), 5xx (server errors), timeouts. For LLM providers you need more:

Metric	Why it matters in AI
HTTP error rate	Standard, same as any service
Timeout rate	If the provider takes longer than the max, consider it a failure
Empty response rate	The provider returns 200 but the content is empty (model refused, content filter)
P95 latency	The provider responds OK but 5× slower than normal → degraded
429 rate (rate limit)	Provider saturation; better to wait than to insist

Example of typical thresholds:

CIRCUIT_THRESHOLDS = {
    "error_rate": 0.5,        # >50% of errors opens the circuit
    "timeout_rate": 0.3,       # >30% of timeouts
    "empty_response_rate": 0.4, # >40% empty responses (rare but it happens)
    "latency_p95_ms": 15000,   # P95 >15s indicates severe degradation
    "rate_limit_rate": 0.3,    # >30% of 429
}
WINDOW_SECONDS = 60            # measurement window
MIN_REQUESTS = 10              # required before evaluating (avoids tripping on 2 errors)
COOLDOWN_SECONDS = 30          # how long OPEN before trying HALF-OPEN

---

Implementation with pybreaker (Python)

pybreaker is the standard library. We extend it for the AI-specific thresholds.

pip install pybreaker

# circuit_breaker.py
import time
import statistics
from collections import deque
import pybreaker
import logging

logger = logging.getLogger(__name__)


class AIServiceCircuit:
    """
    Circuit breaker that considers errors + latency.
    States: closed, open, half-open (delegated to pybreaker).
    """

    def __init__(
        self,
        name: str,
        error_rate_threshold: float = 0.5,
        latency_p95_threshold_ms: int = 15000,
        window_seconds: int = 60,
        min_requests: int = 10,
        cooldown_seconds: int = 30,
    ):
        self.name = name
        self.window_seconds = window_seconds
        self.min_requests = min_requests
        self.error_rate_threshold = error_rate_threshold
        self.latency_p95_threshold_ms = latency_p95_threshold_ms
        self.events: deque = deque(maxlen=1000)  # (timestamp, is_error, latency_ms)

        self.breaker = pybreaker.CircuitBreaker(
            fail_max=999,  # we delegate the decision to our logic
            reset_timeout=cooldown_seconds,
            name=name,
            listeners=[_LogListener()],
        )

    def call(self, func, *args, **kwargs):
        """Wraps func with circuit logic."""
        # Pre-check of health
        if not self._is_healthy():
            self.breaker.fail()  # forces the breaker to be open

        try:
            start = time.perf_counter()
            result = self.breaker.call(func, *args, **kwargs)
            elapsed_ms = int((time.perf_counter() - start) * 1000)
            self._record(False, elapsed_ms)
            return result
        except pybreaker.CircuitBreakerError:
            raise  # circuit is open
        except Exception as e:
            elapsed_ms = int((time.perf_counter() - start) * 1000)
            self._record(True, elapsed_ms)
            raise

    def _record(self, is_error: bool, latency_ms: int):
        now = time.time()
        self.events.append((now, is_error, latency_ms))

    def _is_healthy(self) -> bool:
        now = time.time()
        recent = [e for e in self.events if now - e[0] < self.window_seconds]
        if len(recent) < self.min_requests:
            return True  # not enough data

        errors = sum(1 for _, is_error, _ in recent if is_error)
        error_rate = errors / len(recent)
        if error_rate > self.error_rate_threshold:
            logger.warning(f"Circuit {self.name}: error rate {error_rate:.1%} exceeds threshold")
            return False

        latencies = sorted(latency for _, _, latency in recent)
        p95 = latencies[int(len(latencies) * 0.95)]
        if p95 > self.latency_p95_threshold_ms:
            logger.warning(f"Circuit {self.name}: P95 {p95}ms exceeds {self.latency_p95_threshold_ms}ms")
            return False

        return True

    def state(self) -> str:
        return self.breaker.current_state


class _LogListener(pybreaker.CircuitBreakerListener):
    def state_change(self, cb, old_state, new_state):
        logger.warning(f"Circuit {cb.name}: {old_state.name} → {new_state.name}")

Usage in your worker:

# worker.py
from circuit_breaker import AIServiceCircuit
from openai import OpenAI
import pybreaker

openai_client = OpenAI()
openai_circuit = AIServiceCircuit(
    name="openai",
    error_rate_threshold=0.5,
    latency_p95_threshold_ms=15000,
    cooldown_seconds=30,
)

def call_openai(prompt: str) -> str:
    """Wrapped with a circuit breaker."""
    def _call():
        r = openai_client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": prompt}],
            max_tokens=200,
            timeout=20,
        )
        return r.choices[0].message.content

    try:
        return openai_circuit.call(_call)
    except pybreaker.CircuitBreakerError:
        logger.info("OpenAI circuit open, using fallback")
        return fallback_response(prompt)


def fallback_response(prompt: str) -> str:
    # Cache lookup, alternative model, or degraded message
    return get_cached_response(prompt) or "Degraded service, try again in a few minutes."

---

Designing the fallback

A circuit breaker without a useful fallback = it only postpones the error. Three levels of fallback (from best to worst UX):

Fallback 1 — Another provider

If OpenAI goes down, use Anthropic or Modal. It requires:

• Having the second provider configured and tested
• That your prompt works reasonably well on both
• Assuming your cost goes up (the alternative provider can be more expensive)

def call_with_provider_fallback(prompt: str) -> str:
    try:
        return openai_circuit.call(lambda: call_openai(prompt))
    except pybreaker.CircuitBreakerError:
        try:
            return anthropic_circuit.call(lambda: call_anthropic(prompt))
        except pybreaker.CircuitBreakerError:
            return cache_fallback(prompt)

Fallback 2 — Response cache

Your system caches responses to common queries. When the provider goes down, you serve from the cache.

Pros: instant response, free. Cons: only covers previously seen queries.

def call_with_cache_fallback(prompt: str) -> str:
    try:
        result = openai_circuit.call(lambda: call_openai(prompt))
        cache.set(hash(prompt), result, ttl=3600)
        return result
    except pybreaker.CircuitBreakerError:
        cached = cache.get(hash(prompt))
        if cached:
            return f"[cached response] {cached}"
        return "Service temporarily unavailable."

Fallback 3 — Static degraded response

"Degraded service, try in 5 minutes." Not ideal but better than a 500 error with no context.

def call_with_static_fallback(prompt: str) -> str:
    try:
        return openai_circuit.call(...)
    except pybreaker.CircuitBreakerError:
        return (
            "The assistant is temporarily unavailable. "
            "We're working on restoring it. "
            "Please try again in a few minutes."
        )

Typical combination: a cascade — first the alt provider, then the cache, finally static.

---

Distributing the state across instances

If you have 10 instances and each has its own circuit breaker in memory, they're 10 independent circuits:

• One instance detects degradation → its circuit opens
• The other 9 keep sending to the degraded provider
• Each one eventually detects the degradation, but after spending requests

Solution: distributed state via Redis.

# distributed_circuit.py
import redis

r = redis.Redis(decode_responses=True)

class DistributedCircuit:
    """Circuit state in Redis. All instances read and write it."""

    def __init__(self, name: str, **kwargs):
        self.name = name
        self.key_state = f"circuit:{name}:state"  # "closed", "open", "half-open"
        self.key_events = f"circuit:{name}:events"  # sorted set by timestamp
        # ... rest of the config

    def is_open(self) -> bool:
        state = r.get(self.key_state) or "closed"
        return state == "open"

    def record_event(self, is_error: bool, latency_ms: int):
        now = time.time()
        r.zadd(self.key_events, {f"{now}:{is_error}:{latency_ms}": now})
        r.zremrangebyscore(self.key_events, 0, now - self.window_seconds)  # cleanup
        # ... threshold evaluation here

Trade-off: you add a dependency on Redis for each circuit decision. For many cases it's worth it; for the most latency-sensitive ones, keep it local + periodic sync.

---

Common traps

Trap 1 — Threshold too low, the circuit oscillates. If you open with 3 consecutive errors, a provider blip activates it constantly. Configuring min_requests before evaluating (e.g., 10) and a reasonable time window (60s) avoids false positives.

Trap 2 — Cooldown too short. A 5s cooldown. The provider is still down. Half-open trips, the test fails, it goes back to open. Constant oscillation, no real recovery. Use a longer cooldown (30-120s) with jitter.

Trap 3 — Only measuring HTTP errors. The provider responds 200 OK but with 30s of latency. A classic circuit doesn't detect it. Measure P95 latency too.

Trap 4 — Not having a fallback. Circuit open + no fallback = a 500 error to the user. You only postponed the error by 30s. Design the fallback before implementing the circuit.

Trap 5 — The same circuit for all endpoints. Your chatbot has 3 endpoints: /chat (LLM), /embed (OpenAI embeddings), /moderate (OpenAI moderation). If all 3 share a circuit, a moderation outage affects chat. A circuit per endpoint (or per downstream service).

Trap 6 — Re-entering closed on the first success. Half-open with 1 success → closed. But the provider is still intermittent. Better: require 3-5 successes in a row to confirm recovery.

---

Monitoring the circuit

Mandatory metrics for your dashboard:

• Current state (closed / open / half-open) per circuit, as a gauge
• Time in open (how long we've been rejecting traffic)
• Rate of "circuit open" events (how many times it opened in the last hour)
• Rate of fallback usage (how many requests went to the fallback)
• Rate of successful vs failed recovery (of the half-open tests)

Alert if: the circuit has been open >5min, or it opens >3 times in an hour.

---

Exercise

Your system has:

• OpenAI as the main provider
• Anthropic Claude as the fallback provider
• Redis cache with pre-computed responses for the top-100 queries

Design the circuit breakers + fallback strategy:

1. How many circuits do you configure? What's each one for?
2. What's the fallback cascade?
3. Which thresholds do you use (errors, latency)?
4. What cooldown?
5. How do you notify the team when a circuit opens?

See solution

1. Two circuits:
   • openai_chat: errors, timeouts, P95 latency
   • anthropic_chat: the same, configured for Claude
   • (You don't need a circuit on the cache — it always responds fast)
2. Cascade:
   • Try OpenAI via the openai_chat circuit
   • If the circuit is open: try Anthropic via the anthropic_chat circuit
   • If both circuits are open: cache lookup
   • If cache miss: a static degraded response
3. Thresholds per circuit:
   • error_rate > 50% in the last 60s, minimum 20 requests
   • timeout_rate > 30%
   • latency_p95 > 15s
4. Cooldown: 60s (enough for recovery from blips, not so long that the outage becomes visible if it's a false alarm). With jitter ±15s to avoid a thundering herd when 10 instances retry simultaneously.
5. Notification: Slack webhook + PagerDuty for a circuit open >5min. Slack-only for a circuit open <5min (probable false alarm). Grafana dashboard with the circuits' state in real time for manual inspection.

---

Summary

You learned:

• ✅ The 3-state pattern: closed → open → half-open → closed
• ✅ AI-specific metrics: error rate, timeout rate, P95 latency, empty responses
• ✅ Implementation with pybreaker + an extension for latency
• ✅ Three levels of fallback: another provider, cache, static response
• ✅ Distributing the state across instances via Redis
• ✅ Common traps: bad thresholds, short cooldown, no fallback, the same circuit for everything
• ✅ Mandatory monitoring metrics

Checkpoint: if you can design the complete fallback cascade (provider → alt provider → cache → static) with circuit breakers at the right points, you're ready.

---

Next lesson

06 — Retry strategies and idempotency. Circuit breakers cut the flow to downed downstreams. Retry decides what to do with an individual request that fails. You're going to learn exponential backoff with jitter, correct idempotency keys, and dead letter queues that don't turn into graveyards.

---

Resources

1. Martin Fowler — Circuit Breaker pattern — original explanation.
2. pybreaker GitHub — Python library.
3. Resilience4j (Java) — most mature reference for the pattern, transferable concepts.
4. Hystrix archived (Netflix) — the original, no longer maintained but the principles still hold.
5. Designing Data-Intensive Applications — chapter on faults.