5. Error Recovery and Resilience

Overview

In the previous capsule you configured monitoring and observability: structured logging, LangSmith in production, Prometheus metrics, actionable alerts. Now you know when something fails. But knowing that something failed isn't the same as surviving the failure. An agent in production depends on LLMs that have rate limits, MCP servers that go down, tools that time out, and networks that fluctuate. The question isn't "will it fail?" but "what happens when it does?"

The right answer is never a crash. The right answer is controlled degradation: the system keeps working with reduced capacity, tells the user the result may be partial, and records the failure for later analysis. An agent that returns "I couldn't search the web, but here's my answer based on what I know" is infinitely better than a 500 Internal Server Error.

Connection to the module: This capsule takes capsule 04's monitoring and turns it into action. The alerts tell you "the error rate went up to 15%." Error recovery keeps that 15% from becoming 50%. In capsule 06 you'll see cost control — and the retry policies you implement here have a direct impact on cost (every retry is one more call).

Graceful Degradation

The philosophy: partial beats nothing

Graceful degradation means that when a component fails, the system continues with reduced capacity instead of collapsing. For a multi-agent agent, the degradation hierarchy is:

┌──────────────────────────────────────────────────────────────────┐
│  DEGRADATION LEVELS                                               │
│                                                                  │
│  Level 0: FULL ───────── Everything working                      │
│  │  All agents, tools, MCP servers, the main LLM                 │
│  │                                                               │
│  Level 1: DEGRADED ──── A non-critical component is down        │
│  │  E.g.: the search MCP is down → use a local search tool      │
│  │                                                               │
│  Level 2: MINIMAL ───── Only the essential components            │
│  │  E.g.: the main LLM is down → fall back to a cheaper model   │
│  │                                                               │
│  Level 3: EMERGENCY ─── A guaranteed minimal response            │
│  │  E.g.: everything down → a clear message, request to the DLQ │
└──────────────────────────────────────────────────────────────────┘

Implementing the degradation system

from enum import Enum
from dataclasses import dataclass, field
import logging

logger = logging.getLogger("resilience")


class DegradationLevel(Enum):
    FULL = "full"
    DEGRADED = "degraded"
    MINIMAL = "minimal"
    EMERGENCY = "emergency"


@dataclass
class SystemHealth:
    llm_available: bool = True
    mcp_servers_up: dict[str, bool] = field(default_factory=dict)
    tools_available: dict[str, bool] = field(default_factory=dict)
    fallback_llm_available: bool = True

    @property
    def level(self) -> DegradationLevel:
        if self.llm_available and all(self.mcp_servers_up.values()) and all(self.tools_available.values()):
            return DegradationLevel.FULL
        if not self.llm_available and not self.fallback_llm_available:
            return DegradationLevel.EMERGENCY
        if not self.llm_available or sum(self.tools_available.values()) < len(self.tools_available) * 0.5:
            return DegradationLevel.MINIMAL
        return DegradationLevel.DEGRADED


class GracefulAgent:
    def __init__(self, health: SystemHealth):
        self.health = health

    async def invoke(self, query: str) -> dict:
        level = self.health.level

        if level == DegradationLevel.EMERGENCY:
            return {
                "response": "The service is experiencing problems. "
                            "Please try again in a few minutes.",
                "degradation_level": "emergency",
                "partial": True,
            }

        try:
            if level == DegradationLevel.FULL:
                result = await self._full_invoke(query)
            elif level == DegradationLevel.DEGRADED:
                available = [n for n, up in self.health.tools_available.items() if up]
                result = await self._degraded_invoke(query, available)
            else:
                result = await self._minimal_invoke(query)
            result["degradation_level"] = level.value
            return result
        except Exception as e:
            logger.error(f"Invoke failed at level {level.value}: {e}")
            return {"response": "I couldn't complete your request.", "partial": True}

A silent error is worse than a visible one. If your agent returns a partial result without saying so, the user makes decisions based on incomplete information. Always communicate the degradation level in the response.

Circuit Breakers at the Agent Level

The Circuit Breaker pattern

A circuit breaker works like an electrical breaker: if a component fails repeatedly, you disconnect it temporarily to avoid cascading failures. Without a circuit breaker, every request tries to connect to the downed service, waits for the timeout, fails, and your latency multiplies.

┌──────────────────────────────────────────────────────────────────┐
│  CIRCUIT BREAKER — STATES                                         │
│                                                                  │
│   ┌────────┐   N failures   ┌────────┐   cooldown    ┌─────────┐│
│   │ CLOSED │──────────────→│  OPEN  │──────────────→│HALF-OPEN││
│   │ Normal │               │  Skip  │               │  Test   ││
│   └────┬───┘               └────────┘               └────┬────┘│
│        ↑                   success                        │     │
│        └──────────────────────────────────────────────────┘     │
│                            failure → back to OPEN                │
└──────────────────────────────────────────────────────────────────┘

Implementation

import time
from dataclasses import dataclass
from enum import Enum
from threading import Lock


class CircuitState(Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"


@dataclass
class CircuitBreaker:
    name: str
    failure_threshold: int = 5
    recovery_timeout: float = 60.0

    def __post_init__(self):
        self.state = CircuitState.CLOSED
        self.failure_count = 0
        self.last_failure_time: float = 0
        self._lock = Lock()

    def can_execute(self) -> bool:
        with self._lock:
            if self.state == CircuitState.CLOSED:
                return True
            if self.state == CircuitState.OPEN:
                if time.time() - self.last_failure_time >= self.recovery_timeout:
                    self.state = CircuitState.HALF_OPEN
                    logger.info(f"Circuit {self.name}: OPEN → HALF_OPEN")
                    return True
                return False
            return True  # HALF_OPEN: allow one test call

    def record_success(self):
        with self._lock:
            if self.state == CircuitState.HALF_OPEN:
                self.state = CircuitState.CLOSED
                logger.info(f"Circuit {self.name}: HALF_OPEN → CLOSED")
            self.failure_count = 0

    def record_failure(self):
        with self._lock:
            self.failure_count += 1
            self.last_failure_time = time.time()
            if self.state == CircuitState.HALF_OPEN:
                self.state = CircuitState.OPEN
                return
            if self.failure_count >= self.failure_threshold:
                self.state = CircuitState.OPEN
                logger.warning(f"Circuit {self.name}: CLOSED → OPEN ({self.failure_count} failures)")

    @property
    def status(self) -> dict:
        return {"name": self.name, "state": self.state.value, "failure_count": self.failure_count}

Registry: one breaker per component

Every component that can fail needs its own circuit breaker. A downed MCP server shouldn't open the LLM's circuit:

class CircuitBreakerRegistry:
    def __init__(self):
        self.breakers: dict[str, CircuitBreaker] = {}

    def get_or_create(self, name: str, **kwargs) -> CircuitBreaker:
        if name not in self.breakers:
            self.breakers[name] = CircuitBreaker(name=name, **kwargs)
        return self.breakers[name]

    def get_all_status(self) -> list[dict]:
        return [cb.status for cb in self.breakers.values()]


registry = CircuitBreakerRegistry()
llm_breaker = registry.get_or_create("openai_llm", failure_threshold=3, recovery_timeout=30.0)
mcp_search_breaker = registry.get_or_create("mcp_search", failure_threshold=5, recovery_timeout=60.0)
mcp_files_breaker = registry.get_or_create("mcp_files", failure_threshold=5, recovery_timeout=60.0)

Helper: invoking with a circuit breaker

async def call_with_circuit_breaker(breaker: CircuitBreaker, func, *args, fallback=None, **kwargs):
    if not breaker.can_execute():
        logger.warning(f"Circuit {breaker.name} is OPEN — using fallback")
        if fallback:
            return await fallback(*args, **kwargs) if callable(fallback) else fallback
        raise CircuitOpenError(f"Circuit {breaker.name} is open")
    try:
        result = await func(*args, **kwargs)
        breaker.record_success()
        return result
    except Exception as e:
        breaker.record_failure()
        if fallback:
            return await fallback(*args, **kwargs) if callable(fallback) else fallback
        raise


class CircuitOpenError(Exception):
    pass

Expose /circuits in your API so the dashboard shows which components are in trouble:

@app.get("/circuits")
async def get_circuits():
    return {"circuits": registry.get_all_status()}

Retry Policies

When to retry (and when not to)

Not every error is retryable. Retrying a 400 (invalid input) is pointless. Retrying a 429 (rate limit) with backoff is exactly right:

Implementation with Tenacity

from tenacity import (
    retry, stop_after_attempt, wait_exponential,
    retry_if_exception_type, before_sleep_log,
)
import openai
import httpx

logger = logging.getLogger("retry")

@retry(
    stop=stop_after_attempt(3),
    wait=wait_exponential(multiplier=1, min=2, max=30),
    retry=retry_if_exception_type((
        openai.RateLimitError, openai.APIConnectionError,
        openai.InternalServerError, httpx.ConnectTimeout,
    )),
    before_sleep=before_sleep_log(logger, logging.WARNING),
)
async def invoke_llm_with_retry(llm, messages: list) -> str:
    response = await llm.ainvoke(messages)
    return response.content

If the LLM returns a rate limit, it waits 2s, retries. Fails again: 4s. Third time: 8s. After 3 attempts, it raises the exception.

Differentiated policies per component

Don't use the same policy for everything. An LLM has predictable rate limits; an MCP server can be completely down:

from dataclasses import dataclass

@dataclass
class RetryConfig:
    max_attempts: int
    min_wait: float
    max_wait: float
    retryable_exceptions: tuple

RETRY_CONFIGS = {
    "llm": RetryConfig(3, 2.0, 30.0, (openai.RateLimitError, openai.APIConnectionError, openai.InternalServerError)),
    "mcp_server": RetryConfig(2, 1.0, 10.0, (httpx.ConnectTimeout, httpx.ReadTimeout, ConnectionError)),
    "tool_web": RetryConfig(3, 1.0, 15.0, (httpx.ConnectTimeout, httpx.ReadTimeout)),
    "tool_db": RetryConfig(2, 0.5, 5.0, (ConnectionError, TimeoutError)),
}

def make_retry_decorator(config_name: str):
    c = RETRY_CONFIGS[config_name]
    return retry(
        stop=stop_after_attempt(c.max_attempts),
        wait=wait_exponential(multiplier=1, min=c.min_wait, max=c.max_wait),
        retry=retry_if_exception_type(c.retryable_exceptions),
        before_sleep=before_sleep_log(logger, logging.WARNING),
    )

Retry + Circuit Breaker: the combination

Retry handles transient failures (1-2 failures, then it recovers). A circuit breaker handles persistent failures (the service is down, there's no point continuing). They complement each other:

async def resilient_call(tool_name: str, func, *args, breaker=None, retry_config="tool_web", fallback=None, **kwargs):
    if breaker and not breaker.can_execute():
        logger.warning(f"Circuit {breaker.name} OPEN — skipping {tool_name}")
        return await fallback(*args, **kwargs) if fallback and callable(fallback) else fallback

    decorated = make_retry_decorator(retry_config)(func)
    try:
        result = await decorated(*args, **kwargs)
        if breaker:
            breaker.record_success()
        return result
    except Exception as e:
        if breaker:
            breaker.record_failure()
        logger.error(f"Tool {tool_name} failed after retries: {e}")
        if fallback:
            return await fallback(*args, **kwargs) if callable(fallback) else fallback
        raise

The flow: a request arrives → the circuit breaker checks availability → it executes with the retry policy → if all the retries fail, it records the failure in the circuit breaker → if enough failures accumulate, the circuit opens → subsequent requests jump straight to the fallback.

Fallback Strategies

LLM provider fallback

When your LLM provider has an outage (and it will), you need a plan B. The chain goes from more capable/expensive to less capable/cheap:

from langchain_openai import ChatOpenAI
from langchain_anthropic import ChatAnthropic


class LLMWithFallback:
    def __init__(self):
        self.providers = [
            {"name": "openai_gpt4o", "llm": ChatOpenAI(model="gpt-4o", temperature=0),
             "breaker": registry.get_or_create("openai", failure_threshold=3, recovery_timeout=60)},
            {"name": "anthropic_sonnet", "llm": ChatAnthropic(model="claude-sonnet-4-20250514", temperature=0),
             "breaker": registry.get_or_create("anthropic", failure_threshold=3, recovery_timeout=60)},
            {"name": "openai_mini", "llm": ChatOpenAI(model="gpt-4o-mini", temperature=0),
             "breaker": registry.get_or_create("openai_mini", failure_threshold=5, recovery_timeout=30)},
        ]

    async def ainvoke(self, messages: list) -> dict:
        errors = []
        for provider in self.providers:
            if not provider["breaker"].can_execute():
                continue
            try:
                result = await provider["llm"].ainvoke(messages)
                provider["breaker"].record_success()
                return {"content": result.content, "provider_used": provider["name"],
                        "is_fallback": provider != self.providers[0]}
            except Exception as e:
                provider["breaker"].record_failure()
                errors.append(f"{provider['name']}: {e}")
                logger.warning(f"Provider {provider['name']} failed: {e}")
        raise AllProvidersFailedError(f"All LLM providers failed: {'; '.join(errors)}")

class AllProvidersFailedError(Exception):
    pass

MCP server fallback to local tools

MCP servers are external services — they can go down. For every capability an MCP server exposes, have a local tool that covers the basic case:

async def mcp_search(query: str) -> str:
    breaker = registry.get_or_create("mcp_search")
    if not breaker.can_execute():
        return await local_search_fallback(query)
    try:
        async with httpx.AsyncClient(timeout=10.0) as client:
            resp = await client.post("http://mcp-search:8001/search", json={"query": query})
            resp.raise_for_status()
            breaker.record_success()
            return resp.json()["results"]
    except Exception as e:
        breaker.record_failure()
        logger.warning(f"MCP search failed, using local fallback: {e}")
        return await local_search_fallback(query)

async def local_search_fallback(query: str) -> str:
    try:
        from duckduckgo_search import DDGS
        with DDGS() as ddgs:
            results = list(ddgs.text(query, max_results=5))
            return "\n".join(r["body"] for r in results)
    except Exception:
        return f"Could not search for information about: {query}"

The local tool won't be as good as the full MCP server, but it beats nothing.

Dead Letter Queues

What they are and what they're for

A Dead Letter Queue (DLQ) is where the requests that failed after exhausting every retry and fallback go. They aren't lost — they're stored for later analysis, manual reprocessing, or debugging.

Implementation

import json
import asyncio
import uuid
from dataclasses import dataclass, asdict
from datetime import datetime, timezone
from collections import deque
from pathlib import Path


@dataclass
class DeadLetter:
    id: str
    timestamp: str
    query: str
    user_id: str
    error_type: str
    error_message: str
    retry_count: int
    metadata: dict


class DeadLetterQueue:
    def __init__(self, max_size: int = 10000, persist_path: str = "dlq.jsonl"):
        self.queue: deque[DeadLetter] = deque(maxlen=max_size)
        self.persist_path = Path(persist_path)
        self._lock = asyncio.Lock()

    async def add(self, letter: DeadLetter):
        async with self._lock:
            self.queue.append(letter)
            with open(self.persist_path, "a") as f:
                f.write(json.dumps(asdict(letter), default=str) + "\n")
            logger.error(f"Dead letter added: {letter.id} — {letter.error_type}")

    async def get_stats(self) -> dict:
        if not self.queue:
            return {"total": 0}
        by_type: dict[str, int] = {}
        for dl in self.queue:
            by_type[dl.error_type] = by_type.get(dl.error_type, 0) + 1
        return {"total": len(self.queue), "by_error_type": by_type}

    async def retry_letter(self, letter_id: str, invoke_func) -> dict:
        letter = next((dl for dl in self.queue if dl.id == letter_id), None)
        if not letter:
            return {"error": f"Letter {letter_id} not found"}
        try:
            result = await invoke_func(letter.query)
            self.queue.remove(letter)
            return {"status": "success", "result": result}
        except Exception as e:
            return {"status": "failed_again", "error": str(e)}

dlq = DeadLetterQueue()

Wiring the DLQ into the flow

async def invoke_agent_with_dlq(query: str, user_id: str) -> dict:
    try:
        return await resilient_agent.invoke(query)
    except AllProvidersFailedError as e:
        letter = DeadLetter(
            id=str(uuid.uuid4()),
            timestamp=datetime.now(timezone.utc).isoformat(),
            query=query, user_id=user_id,
            error_type="all_providers_failed", error_message=str(e),
            retry_count=3, metadata={"circuit_states": registry.get_all_status()},
        )
        await dlq.add(letter)
        return {
            "response": "We couldn't process your request. Your query has been "
                        "saved and will be processed when the service recovers.",
            "dead_letter_id": letter.id, "partial": True,
        }

Expose /dlq/stats and /dlq/retry/{id} endpoints for management. If you see 12 dead letters in a day, something's wrong. If you see 0 per week, your resilience system is working.

Error Recovery in Multi-Agent

The Supervisor as recovery manager

In a multi-agent system, the supervisor has an additional responsibility: reassigning tasks when a worker fails.

┌──────────────────────────────────────────────────────────────────┐
│  SUPERVISOR RECOVERY FLOW                                        │
│                                                                  │
│  The Supervisor assigns a task to Agent-A                        │
│       ▼                                                          │
│  Agent-A fails (timeout / error / circuit open)                  │
│       ▼                                                          │
│  The Supervisor evaluates:                                       │
│  ├── Is there an alternative agent? → Reassign to Agent-B        │
│  ├── Is it partially completable? → Continue without that step   │
│  └── Is it critical and unrecoverable? → Escalate to the DLQ     │
└──────────────────────────────────────────────────────────────────┘

Implementing recovery in the Supervisor

from typing import TypedDict, Literal

class AgentState(TypedDict):
    messages: list
    current_task: str
    assigned_agent: str
    retry_count: int
    failed_agents: list[str]
    partial_results: list[dict]
    error_log: list[str]


def supervisor_with_recovery(state: AgentState) -> AgentState:
    failed = state.get("failed_agents", [])
    agent_capabilities = {
        "researcher": ["web_search", "document_analysis"],
        "analyst": ["data_analysis", "document_analysis"],
        "writer": ["content_generation", "summarization"],
        "backup_researcher": ["web_search"],
    }
    available = [
        name for name in agent_capabilities
        if name not in failed and registry.get_or_create(name).can_execute()
    ]
    if not available:
        state["error_log"].append(f"No agents available for: {state['current_task']}")
        return state
    state["assigned_agent"] = available[0]
    return state


def should_retry_or_escalate(state: AgentState) -> Literal["retry", "skip", "dlq"]:
    if state["retry_count"] >= 3:
        return "dlq"

    last_errors = state.get("error_log", [])
    shared_resources = ["llm", "openai", "anthropic", "database"]
    if any(res in err.lower() for err in last_errors for res in shared_resources):
        return "dlq"

    all_agents = ["researcher", "analyst", "writer", "backup_researcher"]
    if all(a in state.get("failed_agents", []) for a in all_agents):
        return "dlq"

    return "retry"

Partial results: don't throw away what already worked

If the researcher completed but the analyst failed, don't discard the researcher's result:

def generate_partial_response(state: AgentState) -> str:
    if not state.get("partial_results"):
        return "The request could not be completed."
    parts = [f"[{pr['agent']}]: {pr['result']}" for pr in state["partial_results"]]
    disclaimer = "\n\n⚠️ This answer may be incomplete. Some components were unavailable."
    return "\n\n".join(parts) + disclaimer

Each worker agent should also have a timeout. If the researcher has gone 45 seconds without responding, cut it off and let the supervisor decide:

import asyncio

async def invoke_agent_with_timeout(agent_func, state: AgentState, timeout_seconds: float = 30.0):
    try:
        return await asyncio.wait_for(agent_func(state), timeout=timeout_seconds)
    except asyncio.TimeoutError:
        logger.warning(f"Agent {state['assigned_agent']} timed out after {timeout_seconds}s")
        return None

Connection to the Project

The Research Agent needs complete resilience to be production-ready:

1. Graceful degradation — 4 defined levels. If the MCP goes down, the agent continues with local tools. If the LLM fails, fall back. If everything fails, an emergency message — never a 500.

2. Circuit breakers — One per component: OpenAI, Anthropic, MCP search, MCP files. A /circuits endpoint to see the state.

3. Retry policies — LLM: 3 retries, 2-30s backoff. MCP: 2 retries, 1-10s. Tools: 3 retries, 1-15s. Never retry 400/401.

4. Fallback chain — GPT-4o → Claude Sonnet → GPT-4o-mini. MCP search → Tavily → DuckDuckGo. If everything fails, an answer based only on the LLM's knowledge.

5. Dead letter queue — Failed requests persisted to disk. Endpoints for monitoring and manual reprocessing.

6. Supervisor recovery — Task reassignment when a worker fails. Accumulated partial results. Escalation to the DLQ when there are no options left.

Troubleshooting

Problem 1: "The retries are doubling my costs"

Likely cause: The retry runs for non-transient errors. A 400 gets retried 3 times — 3 calls that were going to fail anyway.

Solution: Filter by exception type. Only retry transient errors (429, 5xx, timeout, connection). Never retry 400, 401, 403.

Problem 2: "The circuit breaker opens on a transient spike"

Likely cause: A failure_threshold that's too low, or no time window. A burst of 5 failed requests opens the circuit even though the service is fine now.

Solution: Implement a time window: only count failures from the last N seconds. 5 failures in 10 seconds → open. 5 failures in 24 hours → don't open. Reduce recovery_timeout to 15-30 seconds.

Problem 3: "The fallback LLM produces lower-quality answers"

Likely cause: GPT-4o-mini doesn't handle the complex prompts designed for GPT-4o well.

Solution: Adapt the prompt to the model. If you're in fallback, simplify the instruction. Tell the user an alternative model was used.

Problem 4: "The DLQ grows endlessly and nobody reviews it"

Likely cause: There's no review process and no alerts on the DLQ's size.

Solution: Alert when the DLQ goes over a threshold. Automate reprocessing of timeout-type entries (which probably resolve themselves). Purge entries older than 7 days.

Problem 5: "The supervisor reassigns endlessly between agents that fail for the same reason"

Likely cause: All the agents depend on the same downed resource (e.g. they all need the LLM and the LLM is down).

Solution: Before reassigning, check the cause. If it's a shared resource (the LLM, the database), reassigning doesn't help — escalate straight to the DLQ with the failure's context.

Exercises

Exercise 1: Design the degradation chain for your agent

Your Research Agent has: a researcher agent, an analyst agent, a writer agent, an MCP search server, an MCP files server, and OpenAI GPT-4o as the main LLM. Define what happens at each degradation level (FULL → DEGRADED → MINIMAL → EMERGENCY) and which capabilities are lost.

Exercise 2: Configure differentiated retry policies

You have 4 components: the LLM (OpenAI), an MCP search server, a news API, and PostgreSQL. For each one define: max retries, min/max wait, and which errors are retryable vs. not.

Exercise 3: Implement a circuit breaker with a time window

The basic circuit breaker counts absolute failures. The problem: 5 failures in 24 hours isn't an outage; 5 in 10 seconds is. Implement a circuit breaker that only counts failures inside a time window.

from collections import deque
import time

class TimeWindowCircuitBreaker:
    def __init__(self, name: str, failure_threshold: int = 5,
                 window_seconds: float = 60.0, recovery_timeout: float = 30.0):
        self.name = name
        self.failure_threshold = failure_threshold
        self.window_seconds = window_seconds
        self.recovery_timeout = recovery_timeout
        self.failure_timestamps: deque[float] = deque()
        self.state = CircuitState.CLOSED
        self.opened_at: float = 0

    def _clean_old_failures(self):
        cutoff = time.time() - self.window_seconds
        while self.failure_timestamps and self.failure_timestamps[0] < cutoff:
            self.failure_timestamps.popleft()

    def record_failure(self):
        self.failure_timestamps.append(time.time())
        self._clean_old_failures()
        if self.state == CircuitState.HALF_OPEN:
            self.state = CircuitState.OPEN
            self.opened_at = time.time()
            return
        if len(self.failure_timestamps) >= self.failure_threshold:
            self.state = CircuitState.OPEN
            self.opened_at = time.time()

    def record_success(self):
        if self.state == CircuitState.HALF_OPEN:
            self.state = CircuitState.CLOSED
            self.failure_timestamps.clear()

Exercise 4: Design the supervisor's recovery flow

Your supervisor assigns "research AI trends" to the researcher. The researcher fails with a timeout. Draw the decision flow: does it reassign? To whom? What happens if the second one fails? When does it escalate to the DLQ?

Task: "Research AI trends in 2026"
│
├── Attempt 1: researcher
│   └── FAILURE: TimeoutError (30s)
│       ├── A shared-resource error? → NO (an agent timeout)
│       └── Reassign to backup_researcher
│
├── Attempt 2: backup_researcher
│   └── FAILURE: MCP search down
│       ├── Can it work without search? → YES, with a local fallback
│       └── Re-run with DuckDuckGo
│
├── Attempt 3: backup_researcher (with fallback tools)
│   └── PARTIAL SUCCESS → Accumulate the result, continue the pipeline
│
└── If attempt 3 fails:
    ├── retry_count >= 3 → Escalate to the DLQ
    └── Response: "We couldn't complete it. ID: xxx"

Exercise 5: Calculate the impact of retries on cost

Your agent uses GPT-4o ($2.50/M input, $10.00/M output). The average request: 2,000 input tokens, 1,000 output. 500 requests/day. With an 8% error rate and 3 retries per error, how much do the retries cost per month? And with a 20% error rate?

Summary

• Graceful degradation with 4 clear levels (FULL → DEGRADED → MINIMAL → EMERGENCY). The system never crashes — it always returns something useful.
• Circuit breakers per component that prevent cascading failures. If a service is down, jump to the fallback without waiting for a timeout.
• Differentiated retry policies: more retries and longer backoff for LLMs, fewer for databases. Only retry transient errors.
• Chained fallback strategies: the main LLM → an alternative → a cheap one. An MCP server → a local tool. If everything fails, an answer from the model's knowledge.
• Dead letter queues for requests that exhaust every recovery mechanism. They aren't lost — they're saved for analysis and reprocessing.
• Multi-agent recovery where the supervisor reassigns tasks, accumulates partial results, and escalates to the DLQ when there are no options left.

Error recovery isn't about handling exceptions — it's about designing a system that assumes everything can fail and has a plan for every scenario. The goal isn't to avoid failures (impossible) but to survive them gracefully.

Additional Resources

1. Tenacity — Retrying Library — Python's standard library for retrying with backoff and conditions
2. Circuit Breaker — Martin Fowler — The original article defining the pattern
3. Release It! — Michael Nygard — The reference on stability patterns: circuit breakers, bulkheads, timeouts
4. AWS — Exponential Backoff and Jitter — Backoff with jitter to avoid the thundering herd
5. Netflix Hystrix Wiki (archived) — The classic reference on circuit breakers at scale
6. Python asyncio — Timeouts — Official documentation for timeouts with asyncio