3. Scaling and performance

Description

Your agent works. You deployed it with FastAPI, it answers queries, and on your local laptop with one user (you) everything feels reasonable. Now 50 concurrent users show up and the system collapses. Not because the code is wrong — but because an AI agent is not a CRUD endpoint. Every request involves multiple LLM calls (seconds each), tool calls that make network requests, state serialization, and coordination between subagents. Without a scaling strategy, your agent is a single-threaded bottleneck dressed up as a web service.

Scaling agents has unique characteristics that set it apart from traditional web application scaling. A typical REST endpoint responds in 10-50ms. An agent can take 10-60 seconds per request, making 5-15 calls to external APIs along the way. The mental model of "more instances = more throughput" still applies, but you have to solve extra problems: shared state across instances, connections to LLM providers that have rate limits, and caching of responses that are inherently non-deterministic.

Connection to the module: In capsule 02 you deployed the Research Agent with FastAPI. This capsule makes it scalable — async so you don't waste time waiting on I/O, connection pooling so you don't exhaust connections, caching so you don't repeat work, and horizontal scaling to handle real load. In capsule 04, you'll add monitoring to know when to scale.

Async execution: why agents MUST be async

An agent is, fundamentally, a program that waits. It waits for the LLM's response (1-10 seconds). It waits for a tool call's result (0.5-5 seconds). It waits for the checkpoint write to the database (10-100ms). In a typical Research Agent request, the agent spends more than 95% of its time waiting on I/O and less than 5% doing real computation.

If you use synchronous code, while one request waits on GPT-4.1's response, the entire thread is blocked. No other user can be served. With 4 Uvicorn workers, you can only serve 4 simultaneous requests — and each one takes 30+ seconds.

SYNCHRONOUS: 4 workers, 4 simultaneous requests max
─────────────────────────────────────────────────
Worker 1: [████ LLM call ████][██ tool ██][██ LLM ██]  → 30s blocked
Worker 2: [████ LLM call ████][██ tool ██][██ LLM ██]  → 30s blocked
Worker 3: [████ LLM call ████][██ tool ██][██ LLM ██]  → 30s blocked
Worker 4: [████ LLM call ████][██ tool ██][██ LLM ██]  → 30s blocked
Request 5: ⏳ waiting for a free worker...

ASYNC: 1 worker, hundreds of simultaneous requests
─────────────────────────────────────────────────
Event Loop: [req1: LLM] → suspends → [req2: LLM] → suspends → [req3: tool]
            → req1 resolves → [req1: tool] → suspends → [req4: LLM]
            → req2 resolves → [req2: tool] → req3 resolves → ...
            Everything moves forward concurrently.

asyncio with LangGraph

LangGraph exposes async versions of all its execution methods. The rule is simple: use ainvoke instead of invoke, astream instead of stream.

from fastapi import FastAPI
from langgraph.checkpoint.postgres.aio import AsyncPostgresSaver

app = FastAPI()

async def get_graph():
    async with AsyncPostgresSaver.from_conn_string(DATABASE_URL) as checkpointer:
        await checkpointer.setup()
        return research_agent.compile(checkpointer=checkpointer)

@app.post("/research")
async def research(query: str, user_id: str):
    graph = await get_graph()
    config = {"configurable": {"thread_id": f"{user_id}_{uuid4()}"}}

    result = await graph.ainvoke(
        {"messages": [HumanMessage(content=query)]},
        config
    )

    return {"response": result["messages"][-1].content}

Async tools

Your tools have to be async too. A synchronous tool inside an async agent blocks the event loop — cancelling out the entire benefit of async.

import httpx

@tool
async def web_search(query: str) -> str:
    """Search the internet for current information."""
    async with httpx.AsyncClient() as client:
        response = await client.get(
            "https://api.tavily.com/search",
            params={"query": query, "api_key": TAVILY_API_KEY}
        )
        results = response.json()["results"]
        return "\n".join(r["content"] for r in results[:5])

Async antipatterns to avoid

# ❌ BAD: A synchronous call inside an async endpoint
@app.post("/research")
async def research(query: str):
    result = graph.invoke(...)  # synchronous invoke blocks the event loop
    return result

# ❌ BAD: requests (sync) instead of httpx (async)
import requests
@tool
async def search(query: str) -> str:
    response = requests.get(...)  # Blocks the event loop
    return response.text

# ❌ BAD: time.sleep in async code
import time
async def retry_with_backoff():
    time.sleep(5)  # Blocks everything. Use asyncio.sleep(5)

# ✅ CORRECT: If you need legacy sync code, use run_in_executor
import asyncio
@app.post("/research")
async def research(query: str):
    loop = asyncio.get_event_loop()
    result = await loop.run_in_executor(None, sync_heavy_function, query)
    return result

Connection pooling

Every time your agent makes a tool call, it opens a connection — to the database, to an external API, to an MCP server. Without pooling, each request creates and destroys connections. With 50 concurrent requests, you're opening and closing hundreds of connections per minute. Remote servers start rejecting you, the database exhausts its pool, and MCP servers pile up orphaned connections.

Database: AsyncPostgresSaver with a pool

from langgraph.checkpoint.postgres.aio import AsyncPostgresSaver
from sqlalchemy.ext.asyncio import create_async_engine

DB_URL = "postgresql+asyncpg://user:pass@localhost:5432/agents"

engine = create_async_engine(
    DB_URL,
    pool_size=20,
    max_overflow=10,
    pool_timeout=30,
    pool_recycle=1800,
    pool_pre_ping=True,
)

The key parameters:

• pool_size=20 — 20 connections kept permanently open
• max_overflow=10 — Up to 10 extra temporary connections under peak load
• pool_timeout=30 — If no connection is available within 30s, raise an error (better than waiting forever)
• pool_recycle=1800 — Recycle connections every 30 min (avoids stale connections from firewalls/proxies)
• pool_pre_ping=True — Verify the connection is alive before using it

HTTP clients: reuse httpx.AsyncClient

Don't create an AsyncClient per request. Share it:

from contextlib import asynccontextmanager
import httpx

http_client: httpx.AsyncClient | None = None

@asynccontextmanager
async def lifespan(app: FastAPI):
    global http_client
    http_client = httpx.AsyncClient(
        timeout=httpx.Timeout(30.0, connect=5.0),
        limits=httpx.Limits(
            max_connections=100,
            max_keepalive_connections=20,
            keepalive_expiry=30,
        ),
    )
    yield
    await http_client.aclose()

app = FastAPI(lifespan=lifespan)

@tool
async def web_search(query: str) -> str:
    """Search the internet for current information."""
    response = await http_client.get(
        "https://api.tavily.com/search",
        params={"query": query, "api_key": TAVILY_API_KEY}
    )
    return response.text

LLM provider connections

The OpenAI and Anthropic SDKs handle connection pooling internally, but you have to create the client only once:

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="gpt-4.1",
    temperature=0,
    max_retries=3,
    request_timeout=60,
)

Don't call ChatOpenAI(...) inside every request. Create it when the application boots and reuse it. The SDK maintains a pool of HTTPS connections internally.

Without pooling, 50 requests create ~200 ephemeral connections (4 per request) and the destination server starts rejecting them. With pooling, 20 pre-created connections get reused: each request takes one from the pool, uses it, and gives it back. Total connections created: 20 instead of 200.

Caching strategies

An AI agent has a particular usage pattern: many users ask similar questions, tools return the same information if you query them minutes apart, and LLM calls are the most expensive component (in time and money). Caching intelligently reduces latency, cost, and load on external APIs.

What to cache (and what not to)

Caching tool results with Redis

import redis.asyncio as redis
import hashlib
import json

redis_client = redis.from_url("redis://localhost:6379")

def cache_key(tool_name: str, args: dict) -> str:
    """Generate a deterministic key for the tool arguments."""
    args_str = json.dumps(args, sort_keys=True)
    hash_val = hashlib.sha256(args_str.encode()).hexdigest()[:16]
    return f"tool:{tool_name}:{hash_val}"

async def cached_tool_call(tool_name: str, args: dict, ttl: int, func):
    """Wrapper that caches the results of tool calls."""
    key = cache_key(tool_name, args)

    cached = await redis_client.get(key)
    if cached:
        return json.loads(cached)

    result = await func(**args)
    await redis_client.setex(key, ttl, json.dumps(result))
    return result

@tool
async def web_search(query: str) -> str:
    """Search the internet for current information."""
    return await cached_tool_call(
        "web_search",
        {"query": query},
        ttl=600,  # 10 minutes
        func=_raw_web_search
    )

Semantic cache for LLM responses

Exact caching works for tools, but users rarely ask the same question. "What is machine learning?" and "Explain machine learning to me" are semantically the same but textually different. A semantic cache uses embeddings to find similar queries: you generate the embedding of the incoming query, search the cache for entries with cosine similarity > threshold (0.95), and if there's a match you return the cached response.

from langchain_openai import OpenAIEmbeddings
import numpy as np

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

class SemanticCache:
    def __init__(self, similarity_threshold: float = 0.95):
        self.threshold = similarity_threshold
        self.entries: list[dict] = []

    async def get(self, query: str) -> str | None:
        query_embedding = await embeddings.aembed_query(query)
        for entry in self.entries:
            similarity = np.dot(query_embedding, entry["embedding"])
            if similarity >= self.threshold:
                return entry["response"]
        return None

    async def set(self, query: str, response: str, ttl: int = 300):
        embedding = await embeddings.aembed_query(query)
        self.entries.append({
            "query": query, "embedding": embedding,
            "response": response, "expires_at": time.time() + ttl,
        })

Warning: A threshold of 0.95 is conservative. Dropping it to 0.85 increases cache hits but you can end up serving wrong answers to questions that look similar but aren't. Start high and lower it gradually while monitoring quality.

When NOT to cache

1. Conversations with memory — The context is unique to each user and each session
2. Tools with side effects — send_email, create_issue should never be cached
3. Real-time data — If the user asks "what's the price of AAPL right now?", a 10-minute cache returns wrong data
4. Complex research queries — The Research Agent's planning is unique to each query; caching the final result can serve incomplete investigations for questions that only look similar

Horizontal scaling

A single server has limits: CPU, memory, connections, and your LLM provider's rate limit. Horizontal scaling adds more instances of the same service behind a load balancer, spreading the load across them.

The problem: state

The Research Agent uses checkpointing to keep the state of every conversation. With MemorySaver, that state lives in the process's memory. If you have 3 instances and request 1 goes to instance A, request 2 from the same user could land on instance B — which doesn't have the checkpoint.

❌ MemorySaver with multiple instances
─────────────────────────────────────────
                    ┌──────────────┐
     request 1 ────►│ Instance A   │  ← has the checkpoint
                    │ (MemorySaver)│
                    └──────────────┘
Load Balancer
                    ┌──────────────┐
     request 2 ────►│ Instance B   │  ← does NOT have the checkpoint
                    │ (MemorySaver)│
                    └──────────────┘

✅ PostgresSaver with multiple instances
─────────────────────────────────────────
                    ┌──────────────┐
     request 1 ────►│ Instance A   │──┐
                    └──────────────┘  │
Load Balancer                         ▼
                    ┌──────────────┐  ┌──────────┐
     request 2 ────►│ Instance B   │──►│ Postgres │
                    └──────────────┘  │ (shared   │
                    ┌──────────────┐  │ state)     │
     request 3 ────►│ Instance C   │──┘└──────────┘
                    └──────────────┘

Stateless architecture

The rule for horizontal scaling is: the agent instance must not hold local state. All state lives in external services:

from langgraph.checkpoint.postgres.aio import AsyncPostgresSaver

POSTGRES_URL = os.environ["DATABASE_URL"]
REDIS_URL = os.environ["REDIS_URL"]

async def create_graph():
    checkpointer = AsyncPostgresSaver.from_conn_string(POSTGRES_URL)
    await checkpointer.setup()
    return research_agent.compile(checkpointer=checkpointer)

Docker Compose for multiple instances

With Docker Compose you can bring up multiple replicas of the agent using deploy.replicas: 3, each connected to the same PostgreSQL and Redis. An Nginx in front balances requests across the instances. All the state lives in the external services, so any instance can serve any request.

When to scale horizontally

Don't scale prematurely. Measure first:

Performance profiling

Before optimizing, you need to know where the time goes. A Research Agent request passes through multiple phases, each with its own latency:

Anatomy of a 35-second request
──────────────────────────────────────────────────
[2s]   FastAPI routing + validation
[8s]   ████████ LLM call #1 (planning)
[3s]   ███ Tool: web_search
[5s]   █████ Tool: web_search (second query)
[7s]   ███████ LLM call #2 (synthesis)
[0.5s] █ Tool: save_research
[8s]   ████████ LLM call #3 (reflection + final answer)
[1s]   █ Checkpoint write
[0.5s] █ Response serialization
──────────────────────────────────────────────────
Total: 35s
LLM calls: 23s (66%)
Tool calls: 8.5s (24%)
Overhead: 3.5s (10%)

The bottleneck is almost always the LLM. But knowing it with data is different from assuming it.

Instrumenting with timing

import time
from dataclasses import dataclass, field

@dataclass
class RequestProfile:
    request_id: str
    start_time: float = field(default_factory=time.time)
    events: list[dict] = field(default_factory=list)

    def track(self, event_name: str, duration_ms: float, metadata: dict = None):
        self.events.append({
            "event": event_name,
            "duration_ms": round(duration_ms, 2),
            "metadata": metadata or {},
        })

    def summary(self) -> dict:
        total = sum(e["duration_ms"] for e in self.events)
        by_category = {}
        for e in self.events:
            cat = e["event"].split(":")[0]
            by_category[cat] = by_category.get(cat, 0) + e["duration_ms"]

        return {
            "request_id": self.request_id,
            "total_ms": round(total, 2),
            "breakdown": {k: f"{v/total*100:.1f}%" for k, v in by_category.items()},
            "events": self.events,
        }

Using LangSmith for profiling

If you already have LangSmith enabled (capsule 04), every trace shows the timing waterfall — node by node, LLM call by LLM call, tool by tool. Turn tracing on with LANGCHAIN_TRACING_V2=true and LANGCHAIN_PROJECT=research-agent-profiling. It's the most direct tool for spotting bottlenecks without manual instrumentation.

Common optimizations by bottleneck

If the bottleneck is the LLM (>60% of the time):

• Use a faster model for sub-tasks that don't need complex reasoning (gpt-4.1-mini for classification, gpt-4.1 for synthesis)
• Shorten the prompt: shorter prompts = faster responses
• Parallelize independent LLM calls with asyncio.gather

async def parallel_research(queries: list[str]):
    tasks = [llm.ainvoke(q) for q in queries]
    results = await asyncio.gather(*tasks)
    return results

If the bottleneck is the tools (>30% of the time):

• Cache the results (previous section)
• Parallelize independent tool calls
• Use aggressive timeouts with fallbacks

If the bottleneck is the state (>10% of the time):

• Shrink the state: don't store the full message history, use a window
• Use pooled connections to PostgreSQL (previous section)
• Compress the state before serializing

Bottleneck comparison

The 80/20 rule: optimize the LLM calls first. They account for most of the time, and the optimizations are usually the simplest ones (change the model, shorten the prompt, parallelize).

Connection to the project

The Research Agent you deployed in capsule 02 now needs to be scalable:

1. Migrate to async — Replace invoke with ainvoke in every node of the graph. Replace requests with httpx in every tool.
2. PostgresSaver — Replace MemorySaver with AsyncPostgresSaver so the state is shared across instances.
3. Redis cache — Cache web_search results with a 10-minute TTL. Research queries frequently repeat similar searches.
4. Connection pooling — One shared httpx.AsyncClient for all HTTP tool calls. A PostgreSQL connection pool with pool_size=20.
5. Profiling — Use LangSmith to identify which nodes of the Research Agent are the slowest, and optimize those first.

The result: a Research Agent that can handle dozens of concurrent queries without degrading latency, with costs kept down by caching, and ready to scale horizontally when the load calls for it.

Troubleshooting

Problem 1: asyncio.run() cannot be called from a running event loop

Symptom: When you boot the service with Uvicorn, you get RuntimeError: asyncio.run() cannot be called from a running event loop.

Likely cause: You're using asyncio.run() or loop.run_until_complete() inside an async endpoint. Uvicorn already has an event loop running — you can't create another one.

Fix: Use await directly. If you need to call sync code from async, use asyncio.get_event_loop().run_in_executor(None, sync_function). If you're using a library that only has a synchronous API, wrap it with run_in_executor.

Problem 2: connection pool exhausted (QueuePool limit reached)

Symptom: Under load, you get sqlalchemy.exc.TimeoutError: QueuePool limit of X overflow Y reached.

Likely cause: More concurrent requests are asking for connections than the pool can serve. Or connections aren't being returned to the pool (a leak).

Fix: Raise pool_size and max_overflow in proportion to your expected concurrency. Verify you're using async with for sessions (that guarantees the connection goes back to the pool). Check whether you have slow queries holding connections for too long.

engine = create_async_engine(
    DB_URL,
    pool_size=30,       # Up from 20 to 30
    max_overflow=20,     # Up from 10 to 20
    pool_timeout=60,     # More patience before erroring
)

Problem 3: caching serves wrong answers

Symptom: Users report that the agent gives answers that don't match their question, or stale information.

Likely cause: The cache key isn't specific enough (collisions), or the TTL is too long for data that changes often.

Fix: Include more context in the cache key (not just the query, but also the user_id if answers depend on the user). Reduce the TTL. For a semantic cache, raise the similarity_threshold to 0.97+. Add a /cache/clear endpoint to invalidate manually.

Problem 4: LLM provider rate limiting under load

Symptom: Under load, you get 429 Too Many Requests from the LLM provider. The retries cause exponential latency.

Likely cause: Multiple instances of the agent are hitting the same LLM provider, exceeding your tier's rate limit.

Fix: Implement a shared rate limiter (in Redis) that coordinates across instances. Use exponential backoff with jitter. Consider a global request queue. Long term, move up a tier with the provider or spread across multiple providers.

import asyncio
import random

async def call_llm_with_backoff(prompt: str, max_retries: int = 5):
    for attempt in range(max_retries):
        try:
            return await llm.ainvoke(prompt)
        except RateLimitError:
            wait = (2 ** attempt) + random.uniform(0, 1)
            await asyncio.sleep(wait)
    raise Exception("LLM rate limit exceeded after max retries")

Problem 5: memory leak in long-running processes

Symptom: The process's memory grows gradually until it gets killed by OOM (Out Of Memory).

Likely cause: The agent's state accumulates messages without a limit. Every long conversation adds messages to the history and never removes them. It can also be the semantic cache growing unbounded.

Fix: Implement a message window (max 50 messages per conversation, drop the oldest). Add max_entries to the cache with LRU eviction. Monitor the process's memory and set up alerts.

Exercises

Exercise 1: Spot the blocking synchronous code

This endpoint has a performance problem. Find it and fix it:

import requests
import time

@app.post("/analyze")
async def analyze(text: str):
    response = requests.post(
        "https://api.openai.com/v1/chat/completions",
        headers={"Authorization": f"Bearer {API_KEY}"},
        json={"model": "gpt-4.1", "messages": [{"role": "user", "content": text}]}
    )
    time.sleep(1)  # Rate limiting
    return response.json()

import httpx
import asyncio

@app.post("/analyze")
async def analyze(text: str):
    async with httpx.AsyncClient() as client:
        response = await client.post(
            "https://api.openai.com/v1/chat/completions",
            headers={"Authorization": f"Bearer {API_KEY}"},
            json={"model": "gpt-4.1", "messages": [{"role": "user", "content": text}]}
        )
    await asyncio.sleep(1)
    return response.json()

Exercise 2: Design the cache keys

For each tool, design the cache key and decide the TTL:

1. web_search(query="latest AI news")
2. get_stock_price(symbol="AAPL")
3. translate(text="Hello world", target_lang="es")
4. get_weather(city="CDMX")
5. calculate_fibonacci(n=40)

Exercise 3: Diagnose a bottleneck

The following profile shows the timings of a Research Agent request. What's the main bottleneck, and what optimization would you propose?

Event                    Duration
─────────────────────────────────
llm:planning             4,200ms
tool:web_search          2,100ms
tool:web_search          2,300ms
tool:web_search          1,900ms
llm:synthesis            6,500ms
tool:save_research         150ms
llm:reflection           5,800ms
checkpoint:write           200ms
─────────────────────────────────
Total:                  23,150ms

Exercise 4: A horizontal scaling decision

You have a Research Agent deployed with 2 instances. The metrics show:

• Average CPU: 15%
• Memory: 40%
• p50 latency: 12s, p95: 45s, p99: 120s
• OpenAI rate limit errors: 15/hour
• Queued requests: 0

Do you need more instances? What would you do?

Exercise 5: Design the cache architecture

Design the complete caching strategy for the Research Agent. Define: which components you cache, which backend you use, what TTL, and how you invalidate the cache.

@app.delete("/cache")
async def clear_cache(tool: str | None = None):
    if tool:
        keys = await redis_client.keys(f"tool:{tool}:*")
        if keys:
            await redis_client.delete(*keys)
    else:
        await redis_client.flushdb()
    return {"status": "cleared"}

Summary

• Async execution is mandatory for agents. An agent spends >95% of its time waiting on I/O (LLM calls, tool calls, DB writes). Without async, a worker serves one request at a time. With async, a single event loop handles hundreds of concurrent requests. Use ainvoke, httpx, and asyncio.sleep — never their synchronous counterparts.
• Connection pooling prevents resource exhaustion. Reuse connections to PostgreSQL (pool_size), to HTTP APIs (a shared httpx.AsyncClient), and to Redis. Without pooling, 50 concurrent requests create hundreds of ephemeral connections that saturate servers.
• Caching reduces latency, cost, and load on external APIs. Cache tool results with a TTL appropriate to the type of data. Consider a semantic cache for LLM responses (with a high threshold, 0.95+). Never cache side effects or real-time data.
• Horizontal scaling requires stateless agents. Replace MemorySaver with AsyncPostgresSaver. All state in external services (PostgreSQL, Redis). Any instance can serve any request.
• The main bottleneck is the LLM calls (50-70% of the time). Optimize first with faster models for sub-tasks, parallelizing independent calls, and shortening prompts. Tools are the second bottleneck, solved with caching and parallelization.
• LLM provider rate limits are not solved by more instances. More instances = more requests = more rate limits. The fixes: a proactive rate limiter, caching, model routing, a tier upgrade.
• Measure before you optimize. Use LangSmith or custom instrumentation to profile every request. Find out where the time goes before you change any code.

Next capsule: Monitoring and Observability — you'll set up structured logging, LangSmith in production, actionable alerts, and dashboards so you know when your agent needs attention.

Additional resources

1. FastAPI Async Documentation — The official guide to async/await in FastAPI
2. LangGraph Async API — Documentation for ainvoke, astream, and async checkpointers
3. Redis Caching Patterns — Caching patterns with Redis: TTL, LRU, pub/sub
4. SQLAlchemy Async Engine — Connection pooling and async sessions
5. httpx Documentation — Async HTTP client for Python, the replacement for requests
6. Python asyncio Documentation — Official reference for the event loop, tasks, and gathering