Module 3: Function Calling Patterns
7. Retry Patterns and Circuit Breakers
Overview
Your agent works perfectly in the notebook. It calls Tavily, gets results, the LLM synthesizes an answer. You ship it to production on a Monday. Tuesday at 3am, Tavily has a latency spike. Your agent calls, there's no response in 30 seconds, Python raises a TimeoutError, your agent loop crashes, the user sees "Internal Server Error." An agent that doesn't handle failures isn't a production agent — it's a demo.
This capsule teaches you the resilience patterns that separate demo agents from production-ready ones. Retry with exponential backoff for transient errors. Circuit breakers so you stop hammering a downed service. Fallback tools so you have a plan B. Per-tool timeouts so one slow service doesn't block everything. And finally, how to combine all these patterns into a reusable wrapper.
In Module 2 (capsule 06) you saw basic error handling: try/except, descriptive messages, input validation. Here we go to the next level: resilience patterns from microservices engineering (Netflix, AWS, Google) adapted to AI agents. The mindset shifts from "what do I do if it fails" to "how do I design a system that expects failures and recovers automatically."
The Problem: APIs That Fail
The math of failures
When an API says "99.9% uptime", it sounds excellent. But let's do the math:
99.9% uptime = 0.1% downtime = 8.76 hours per year
Your agent in production:
- 100 requests/hour × 4 tool calls on average = 400 API calls/hour
- 400 × 24 = 9,600 API calls/day
- With 99.9% uptime: ~10 calls will fail every day
The question isn't "will it fail?" but "does my agent know what to do when it does?"
Types of failures
| Type | Example | Transient? | Correct action |
|---|---|---|---|
| Timeout | API doesn't respond in 10s | Yes | Retry with backoff |
| Rate limit (429) | "Too Many Requests" | Yes | Retry with a long delay |
| Server error (500) | Bug on the service's side | Maybe | Retry 1-2 times, then fallback |
| Network error | DNS failure, connection reset | Yes | Retry with backoff |
| Auth error (401/403) | Expired API key | No | No retry — fix the config |
| Not found (404) | Resource doesn't exist | No | No retry — return the error to the model |
| Service down | API completely down | Temporary | Circuit breaker + fallback |
Golden rule: retry transient errors, fail fast on permanent ones.
What happens without resilience patterns
from langchain_core.tools import tool
import random
@tool
def search_web(query: str) -> str:
"""Search the web for information."""
if random.random() < 0.3:
raise ConnectionError("API temporarily unavailable")
return f"Results for: {query}"
try:
result = search_web.invoke({"query": "AI agents 2026"})
except ConnectionError as e:
print(f"CRASH: {e}")
# No retry. No fallback. The agent stopped.
Retry with Exponential Backoff
Why "just try again" isn't enough
- Immediate retry: if the API is saturated, hammering it saturates it more
- Retry with a fixed delay: if 100 clients retry at exactly 5s, you create a thundering herd
- Retry with exponential backoff: each attempt waits longer — 1s, 2s, 4s. It spreads the load
- Backoff + jitter: adds a random component so not everyone retries at the same moment
Implementation from scratch
import time, random
from functools import wraps
def retry_with_backoff(max_retries=3, base_delay=1.0, max_delay=60.0,
exponential_base=2.0, jitter=True,
retryable_exceptions=(Exception,)):
"""Decorator that adds retry with exponential backoff."""
def decorator(func):
@wraps(func)
def wrapper(*args, **kwargs):
for attempt in range(max_retries + 1):
try:
return func(*args, **kwargs)
except retryable_exceptions as e:
if attempt == max_retries:
raise
delay = min(base_delay * (exponential_base ** attempt), max_delay)
if jitter:
delay = delay * (0.5 + random.random())
print(f" ⟳ Attempt {attempt+1} failed: {e}. Retrying in {delay:.1f}s...")
time.sleep(delay)
return wrapper
return decorator
Using it with tools
from langchain_core.tools import tool
import time, random
@retry_with_backoff(max_retries=3, base_delay=1.0, retryable_exceptions=(ConnectionError, TimeoutError))
def _search_impl(query: str) -> str:
if random.random() < 0.5:
raise ConnectionError("Service temporarily unavailable")
return f"Results for: {query}"
@tool
def search_web(query: str) -> str:
"""Search the web with automatic retry."""
return _search_impl(query)
# ⟳ Attempt 1 failed: ... Retrying in 1.2s...
# ⟳ Attempt 2 failed: ... Retrying in 2.7s...
# Results for: LangChain agents ← third attempt succeeded
Using tenacity (the standard library for production)
In production, you don't need to write your own decorator. tenacity is Python's library for retry logic:
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type
@retry(
stop=stop_after_attempt(4),
wait=wait_exponential(multiplier=1, min=1, max=30),
retry=retry_if_exception_type((ConnectionError, TimeoutError)),
)
def call_external_api(query: str) -> str:
"""Call an external API with retry via tenacity."""
import httpx
response = httpx.get("https://api.example.com/search", params={"q": query}, timeout=10.0)
response.raise_for_status()
return response.text
When NOT to retry
If the same request with the same parameters could work 5 minutes later → retry. If not → fail fast. 401, 404, 400 errors and validation errors are not retryable.
Circuit Breaker Pattern
The problem retry doesn't solve
Tavily's API is completely down. Your retry makes 4 attempts with backoff: 1s, 2s, 4s = 7 seconds per request. With 50 simultaneous users: 50 × 7s = 350 seconds of wasted CPU, plus 200 requests to an API that's already suffering.
Retry helps with transient failures. When a service is consistently down, retry makes it worse. The circuit breaker says "this service has failed N times in a row — stop trying for a while."
State machine: CLOSED → OPEN → HALF_OPEN
┌─────────────────────────────────────────────┐
│ │
▼ │
┌────────┐ N consecutive ┌────────┐ timeout │
│ CLOSED │ ──────────▶ │ OPEN │ ─────────▶│
│(normal)│ failures │(stopped)│ expired │
└────────┘ └────────┘ │
▲ │
│ success ┌───────────┐ │
└────────────────── │ HALF_OPEN │ ◀──────────┘
│ (testing) │
└───────────┘
│ failure → back to OPEN
- CLOSED (normal): calls pass through as usual
- OPEN (tripped): calls fail immediately — it protects the downed service
- HALF_OPEN (testing): allows one test call to see whether it recovered
Complete implementation
import time
import threading
class CircuitBreakerOpen(Exception):
pass
class CircuitBreaker:
def __init__(self, failure_threshold=3, reset_timeout=60.0, name="default"):
self.failure_threshold = failure_threshold
self.reset_timeout = reset_timeout
self.name = name
self.failure_count = 0
self.state = "CLOSED" # CLOSED, OPEN, HALF_OPEN
self.last_failure_time = None
self._lock = threading.Lock()
def call(self, func, *args, **kwargs):
with self._lock:
if self.state == "OPEN":
if self.last_failure_time and (time.time() - self.last_failure_time) > self.reset_timeout:
self.state = "HALF_OPEN"
else:
raise CircuitBreakerOpen(f"Circuit breaker '{self.name}' is OPEN")
try:
result = func(*args, **kwargs)
except Exception:
self._on_failure()
raise
else:
self._on_success()
return result
def _on_success(self):
with self._lock:
if self.state == "HALF_OPEN":
self.state = "CLOSED"
self.failure_count = 0
def _on_failure(self):
with self._lock:
self.failure_count += 1
self.last_failure_time = time.time()
if self.state == "HALF_OPEN":
self.state = "OPEN"
elif self.failure_count >= self.failure_threshold:
self.state = "OPEN"
Using the circuit breaker
import random, time
breaker = CircuitBreaker(failure_threshold=3, reset_timeout=10, name="search_api")
def unreliable_search(query: str) -> str:
if random.random() < 0.7:
raise ConnectionError("Service unavailable")
return f"Results for: {query}"
for i in range(8):
try:
result = breaker.call(unreliable_search, f"query_{i}")
print(f" Request {i}: {result}")
except CircuitBreakerOpen as e:
print(f" Request {i}: BLOCKED — {e}")
except ConnectionError as e:
print(f" Request {i}: FAILED — {e}")
time.sleep(0.5)
# First 3: FAILED → the breaker opens
# Next ones: BLOCKED instantly (it doesn't even try)
# After reset_timeout: HALF_OPEN → test → if success: CLOSED
Trade-off
| Without a circuit breaker | With a circuit breaker |
|---|---|
| 50 requests × 7s timeout = 350s wasted | 3 requests fail, 47 fail instantly |
| The downed API gets 50 requests | The downed API gets only 3 |
| The user waits 7s per request | The user sees an immediate error → fallback |
Fallback Tools
When the primary tool fails, have a plan B
Retry and the circuit breaker deal with the same API. But for many use cases there are alternative services. If Tavily is down, DuckDuckGo. If OpenWeatherMap doesn't respond, Open-Meteo.
from langchain_core.tools import tool
def with_fallbacks(*funcs):
"""Run functions in order until one works."""
def execute(**kwargs) -> str:
errors = []
for i, func in enumerate(funcs):
try:
result = func(**kwargs)
if i > 0: print(f" ↪ Fallback #{i} succeeded: {func.__name__}")
return result
except Exception as e:
errors.append(f"{func.__name__}: {e}")
return f"Every provider failed: {'; '.join(errors)}"
return execute
def tavily_search(query: str) -> str:
import random
if random.random() < 0.6: raise ConnectionError("Tavily unavailable")
return f"[Tavily] Results for: {query}"
def duckduckgo_search(query: str) -> str:
import random
if random.random() < 0.3: raise ConnectionError("DuckDuckGo timeout")
return f"[DuckDuckGo] Results for: {query}"
def cached_search(query: str) -> str:
return f"[Cache] Previous results for: {query} (they may not be up to date)"
search_fallback = with_fallbacks(tavily_search, duckduckgo_search, cached_search)
@tool
def resilient_search(query: str) -> str:
"""Search the web with automatic fallbacks."""
return search_fallback(query=query)
Fallback with a quality indicator
When you use a fallback, the quality may be lower. The model should know. Add a prefix like [quality=FRESH] or [quality=CACHED] to the result. The model gets [quality=CACHED] and can say: "According to previous data (which may not be fully up to date)..."
Common fallback pairs for agents
| Primary tool | Fallback | Case |
|---|---|---|
| Tavily Search | DuckDuckGo / Brave | Web search |
| OpenWeatherMap | Open-Meteo | Weather |
| Google Maps | Nominatim (OSM) | Geocoding |
| GPT-4.1 | GPT-4.1-mini | LLM (lower quality, cheaper) |
| Live DB | Redis cache | Data that rarely changes |
Timeout per Tool
Every tool needs its own timeout
A calculator solves 2+2 in microseconds. A search_web needs 2-5s. An analyze_document can take 30s. If you use the same timeout for all of them, it's either too short for the slow ones or too long to detect that the fast ones hung.
import concurrent.futures
TOOL_TIMEOUTS = {"calculator": 2.0, "search_web": 10.0, "analyze_document": 45.0, "get_weather": 8.0}
DEFAULT_TIMEOUT = 10.0
def execute_with_timeout(tool_fn, args: dict, tool_name: str) -> str:
timeout = TOOL_TIMEOUTS.get(tool_name, DEFAULT_TIMEOUT)
with concurrent.futures.ThreadPoolExecutor(max_workers=1) as executor:
future = executor.submit(tool_fn.invoke, args)
try:
return str(future.result(timeout=timeout))
except concurrent.futures.TimeoutError:
future.cancel()
return f"Error: {tool_name} exceeded its {timeout}s timeout."
Rule for picking timeouts: timeout = 2-3× the normal expected time. Calculator: 1-2s. Web search: 8-12s. Document analysis: 30-60s.
Combining Patterns: Resilient Tool Wrapper
A wrapper that composes everything
In production you use retry + circuit breaker + timeout + fallback together. If you add them by hand to every tool, your code turns into spaghetti. The solution:
import time, random, concurrent.futures
from typing import Callable, Optional
class ResilientToolWrapper:
"""Wrapper: retry + circuit breaker + timeout + fallback."""
def __init__(self, primary_fn: Callable, fallback_fn: Optional[Callable] = None,
max_retries=3, base_delay=1.0, timeout=10.0,
circuit_failure_threshold=5, circuit_reset_timeout=60.0,
name="tool", retryable_exceptions=(ConnectionError, TimeoutError, OSError)):
self.primary_fn = primary_fn
self.fallback_fn = fallback_fn
self.max_retries = max_retries
self.base_delay = base_delay
self.timeout = timeout
self.name = name
self.retryable_exceptions = retryable_exceptions
self.breaker = CircuitBreaker(circuit_failure_threshold, circuit_reset_timeout, name)
self.metrics = {"total_calls": 0, "successes": 0, "retries": 0,
"fallbacks_used": 0, "circuit_breaks": 0, "timeouts": 0}
def _with_timeout(self, fn, *args, **kwargs):
with concurrent.futures.ThreadPoolExecutor(max_workers=1) as ex:
future = ex.submit(fn, *args, **kwargs)
try:
return future.result(timeout=self.timeout)
except concurrent.futures.TimeoutError:
self.metrics["timeouts"] += 1
raise TimeoutError(f"{self.name} exceeded {self.timeout}s timeout")
def __call__(self, *args, **kwargs):
self.metrics["total_calls"] += 1
last_error = None
for attempt in range(self.max_retries + 1):
try:
result = self.breaker.call(self._with_timeout, self.primary_fn, *args, **kwargs)
self.metrics["successes"] += 1
return result
except CircuitBreakerOpen:
self.metrics["circuit_breaks"] += 1
last_error = "Circuit breaker OPEN"
break
except self.retryable_exceptions as e:
last_error = e
if attempt < self.max_retries:
self.metrics["retries"] += 1
time.sleep(self.base_delay * (2 ** attempt) * (0.5 + random.random()))
if self.fallback_fn is not None:
try:
self.metrics["fallbacks_used"] += 1
return f"[fallback] {self.fallback_fn(*args, **kwargs)}"
except Exception as fb_err:
return f"Error: primary ({last_error}) and fallback ({fb_err}) both failed."
return f"Error after {self.max_retries + 1} attempts: {last_error}"
Using it with a LangChain tool
from langchain_core.tools import tool
def primary_search(query: str) -> str:
if random.random() < 0.4:
raise ConnectionError("Tavily temporarily unavailable")
time.sleep(1)
return f"[Tavily] 5 results for '{query}'"
def backup_search(query: str) -> str:
time.sleep(0.5)
return f"[DuckDuckGo] 3 results for '{query}'"
search_wrapper = ResilientToolWrapper(
primary_fn=primary_search, fallback_fn=backup_search,
max_retries=2, base_delay=0.5, timeout=5.0,
circuit_failure_threshold=4, circuit_reset_timeout=30.0,
name="web_search",
)
@tool
def web_search(query: str) -> str:
"""Search the web with retry, circuit breaker, timeout and fallback."""
return search_wrapper(query=query)
The complete flow
Request ──▶ Circuit Breaker: OPEN?
│ Yes → fail fast
│ No ↓
Timeout: run with a limit
│ Exceeded → TimeoutError
│ OK → return ✓
↓ (if it fails)
Retry: retryable + attempts left?
│ Yes → retry with backoff
│ No ↓
Fallback: is there an alternative?
│ Yes → run the fallback
│ No → return the error to the model
When to Use Each Pattern
Decision table
| Pattern | Problem it solves | When to use it | When NOT to |
|---|---|---|---|
| Retry + backoff | Transient errors | An API with sporadic failures | Permanent errors (auth, 404) |
| Circuit breaker | Prolonged service outage | APIs with frequent downtime | Local tools that never fail |
| Fallback | Dependence on one provider | There are alternatives | No viable alternative exists |
| Timeout | Tools that hang | Every tool in production | Ultra-fast tools (<50ms) |
| All combined | Full resilience | Critical tools in production | A prototype or demo |
Decision framework
Can the tool fail?
├── No (local computation) → No patterns (maybe a timeout for safety)
└── Yes
├── Are the failures transient? → Retry with backoff
├── Can it be down for a long stretch? → + Circuit Breaker
├── Is there an alternative provider? → + Fallback
└── Is it production with real users? → ALL the patterns
Trade-offs
| Pattern | Benefit | Cost |
|---|---|---|
| Retry | Recovers from transient errors | Higher latency on failure |
| Circuit Breaker | Fail fast, protects the service | Can block valid requests |
| Fallback | Always available | Lower quality, maintaining 2+ providers |
| Timeout | Prevents hangs | Can cut off legitimately slow operations |
Connection to the Project
In this module's project (capsule 08), the resilience patterns apply directly:
- The extraction tools call external APIs → retry + backoff when the LLM provider has latency
- Routing sends entities to specialized processors → circuit breaker if a processor fails repeatedly
- The system needs to work 24/7 → fallbacks for search and processing
- Each processor has different latency → differentiated timeouts
In the evolving project (M4-10):
- M4: StateGraph nodes with retry + circuit breaker
- M5: If a tool fails during planning, the planner re-plans
- M8: Each agent in multi-agent systems is resilient independently
- M10: In production, circuit breaker metrics go to the monitoring dashboard
Troubleshooting
Problem 1: "The retry takes too long and the user gets impatient"
Cause: 3+ retries with long backoff in a user-facing context.
Solution: For interactive UX, cut it down to 2 retries with short delays (0.5s, 1s). Add a total timeout that caps the time including every retry (e.g. 8s max).
Problem 2: "The circuit breaker opens on a momentary spike"
Cause: 3 quick failures open the breaker, but the service already recovered.
Solution: Use a time window — only count failures from the last N seconds. Store failure timestamps in a list and filter out the ones outside the window before evaluating the threshold. That way a spike of 3 errors in 1 second doesn't open the breaker if it normally works fine.
Problem 3: "I don't know which errors are retryable"
Solution: Status 429, 500, 502, 503, 504 are retryable. ConnectionError, TimeoutError, OSError too. 401, 404, 400 errors and validation errors: no retry.
Problem 4: "The fallback returns stale data and the model doesn't know"
Solution: Always include quality metadata in the result: [source=tavily, freshness=live] for fresh data, [source=cache, freshness=24h old] for cache. The model can then communicate freshness to the user.
Exercises
Exercise 1: Retry decorator with logging (Easy)
Create a @retry_logged decorator that: (1) retries up to 3 times with 1s, 2s, 4s backoff, (2) prints each attempt with a relative timestamp, (3) returns the result or raises the last exception. Test it with a function that fails the first 2 times.
See solution
import time
from functools import wraps
def retry_logged(max_retries=3, base_delay=1.0):
def decorator(func):
@wraps(func)
def wrapper(*args, **kwargs):
t0 = time.time()
for attempt in range(max_retries + 1):
try:
print(f"[{time.time()-t0:.1f}s] Attempt {attempt+1}: SUCCESS")
return func(*args, **kwargs)
except Exception as e:
print(f"[{time.time()-t0:.1f}s] Attempt {attempt+1}: FAILED ({e})")
if attempt == max_retries: raise
time.sleep(base_delay * (2 ** attempt))
return wrapper
return decorator
call_count = 0
@retry_logged(max_retries=3, base_delay=0.5)
def flaky_api(query: str) -> str:
global call_count
call_count += 1
if call_count <= 2: raise ConnectionError("Temporarily unavailable")
return f"Result: {query}"
call_count = 0
print(flaky_api("test"))
Exercise 2: Circuit breaker with statistics (Medium)
Implement a simplified circuit breaker that: (1) opens after 3 failures, (2) has a reset_timeout of 5s, (3) exposes stats() with {"state", "failures", "total_calls", "blocked_calls"}. Simulate 10 calls where the first 4 fail.
See solution
import time
class SimpleCircuitBreaker:
def __init__(self, threshold=3, reset_timeout=5.0):
self.threshold = threshold
self.reset_timeout = reset_timeout
self.failures = 0
self.state = "CLOSED"
self.last_failure_time = None
self.total_calls = 0
self.blocked_calls = 0
def call(self, func, *args, **kwargs):
self.total_calls += 1
if self.state == "OPEN":
if time.time() - self.last_failure_time > self.reset_timeout:
self.state = "HALF_OPEN"
else:
self.blocked_calls += 1
raise Exception("OPEN — blocked")
try:
result = func(*args, **kwargs)
if self.state == "HALF_OPEN":
self.state = "CLOSED"
self.failures = 0
return result
except Exception:
self.failures += 1
self.last_failure_time = time.time()
if self.failures >= self.threshold:
self.state = "OPEN"
raise
def stats(self):
return {"state": self.state, "failures": self.failures,
"total_calls": self.total_calls, "blocked_calls": self.blocked_calls}
cb = SimpleCircuitBreaker(threshold=3, reset_timeout=5.0)
for i in range(10):
try:
result = cb.call(lambda n=i: (_ for _ in ()).throw(ConnectionError(f"Fail {n}")) if n < 4 else f"OK {n}")
print(f" [{i}] {result}")
except Exception as e:
print(f" [{i}] ERROR: {e}")
if i == 6:
time.sleep(6)
print(f"Stats: {cb.stats()}")
Exercise 3: Fallback chain with metrics (Medium)
Create fallback_chain that: (1) takes a list of (name, func) tuples, (2) runs them in order until one succeeds, (3) returns {"result", "provider_used", "providers_tried", "total_time"}. Test it with 3 providers: the first always fails, the second fails 50% of the time, the third always works.
See solution
import time, random
def fallback_chain(providers, **kwargs):
t0, tried = time.time(), []
for name, func in providers:
tried.append(name)
try:
return {"result": func(**kwargs), "provider_used": name,
"providers_tried": tried, "total_time": round(time.time()-t0, 3)}
except Exception:
continue
return {"result": None, "provider_used": None,
"providers_tried": tried, "total_time": round(time.time()-t0, 3)}
providers = [
("always_fail", lambda query: (_ for _ in ()).throw(ConnectionError("down"))),
("sometimes", lambda query: "OK" if random.random() > 0.5 else (_ for _ in ()).throw(ConnectionError("50/50"))),
("always_ok", lambda query: f"[backup] {query}"),
]
for i in range(5):
out = fallback_chain(providers, query=f"test_{i}")
print(f" → {out['provider_used']}: tried {out['providers_tried']}")
Exercise 4: Resilient tool with the complete wrapper (Hard)
Using ResilientToolWrapper, create a @tool called smart_search with: a primary (40% fail), a backup as the fallback, retry=2, timeout=3s, circuit breaker threshold=4. Run 12 requests and show the final metrics.
See solution
from langchain_core.tools import tool
import time, random
wrapper = ResilientToolWrapper(
primary_fn=lambda query: (_ for _ in ()).throw(ConnectionError("down")) if random.random() < 0.4 else f"[Primary] {query}",
fallback_fn=lambda query: f"[Backup] {query}",
max_retries=2, base_delay=0.3, timeout=3.0,
circuit_failure_threshold=4, circuit_reset_timeout=15.0, name="smart_search",
)
@tool
def smart_search(query: str) -> str:
"""Resilient search."""
return wrapper(query=query)
for i in range(12):
print(f" [{i:2d}] {smart_search.invoke({'query': f'q{i}'})[:60]}")
print(f"\nMetrics: {wrapper.metrics}")
Exercise 5: Agent loop with resilient tools in parallel (Hard)
Build an agent loop that: (1) has 2 tools (get_weather, search_web), each with its own ResilientToolWrapper, (2) runs tool calls in parallel with ThreadPoolExecutor, (3) returns the answer + metrics from both wrappers.
See solution
from concurrent.futures import ThreadPoolExecutor, as_completed
from langchain.chat_models import init_chat_model
from langchain_core.tools import tool
from langchain_core.messages import HumanMessage, ToolMessage
w_weather = ResilientToolWrapper(
lambda city: f"{city}: 22°C", lambda city: f"{city}: ~20°C (cached)",
max_retries=2, timeout=5.0, circuit_failure_threshold=3, name="weather")
w_search = ResilientToolWrapper(
lambda query: f"5 results for '{query}'", lambda query: f"Basic results for '{query}'",
max_retries=2, timeout=8.0, circuit_failure_threshold=3, name="search")
@tool
def get_weather(city: str) -> str:
"""Resilient weather."""
return w_weather(city=city)
@tool
def search_web(query: str) -> str:
"""Resilient search."""
return w_search(query=query)
all_tools = [get_weather, search_web]
tbn = {t.name: t for t in all_tools}
mwt = init_chat_model("openai:gpt-4.1-mini").bind_tools(all_tools)
def resilient_loop(user_input, max_iter=5):
messages = [HumanMessage(content=user_input)]
for _ in range(max_iter):
resp = mwt.invoke(messages)
messages.append(resp)
if not resp.tool_calls:
return {"response": resp.content, "metrics": {
"weather": w_weather.metrics, "search": w_search.metrics}}
with ThreadPoolExecutor(max_workers=len(resp.tool_calls)) as ex:
futs = {ex.submit(tbn[tc["name"]].invoke, tc["args"]): tc
for tc in resp.tool_calls if tc["name"] in tbn}
for f in as_completed(futs):
tc = futs[f]
try: result = str(f.result(timeout=15))
except Exception as e: result = f"Error: {e}"
messages.append(ToolMessage(content=result, tool_call_id=tc["id"]))
return {"response": "Max iterations", "metrics": {
"weather": w_weather.metrics, "search": w_search.metrics}}
Summary
In this capsule you learned:
- APIs fail — 99.9% uptime = 8.7 hours/year of downtime. With hundreds of daily tool calls, failures are a certainty, not a possibility
- Retry with exponential backoff handles transient errors: each attempt waits longer (1s, 2s, 4s) and jitter avoids thundering herds
- The circuit breaker protects downed services: after N failures, it stops trying (CLOSED → OPEN → HALF_OPEN). Fail fast instead of waiting on useless timeouts
- Fallback tools give you a plan B: Tavily → DuckDuckGo, live API → cache. Always with a quality indicator so the model knows what it's presenting
- Timeout per tool prevents hangs: calculator=1s, search=10s, analysis=45s. Timeout = 2-3× the normal time
- ResilientToolWrapper composes every pattern: retry → circuit breaker → timeout → fallback, with metrics
- The central trade-off: more resilience = more complexity. For production it's worth it; for prototypes, basic retry is enough
Next capsule: Module Project — an extraction + routing system that combines parallel function calling, structured extraction, tool routing, and the resilience patterns from this capsule.
Additional Resources
- Tenacity — Python's standard library for retry logic with decorators
- Circuit Breaker — Martin Fowler — The pattern's original article
- Microsoft — Retry Pattern — Azure Architecture guide on retry patterns
- Microsoft — Circuit Breaker Pattern — Detailed guide with state diagrams
- AWS — Exponential Backoff and Jitter — Analysis of backoff strategies
- Netflix Hystrix — The library that popularized circuit breakers in microservices