Module 8: Project The Reservo Agent In Production

What Your Agent Still Needs

Next →

Description

The six previous lessons delivered an operated agent: instrumented with trace_id and structured logging (M2), measured in cost and modeled latency (M3/M4), gated against form regressions with real evidence (M5), and hardened with a circuit breaker remembering across runs (M6). This lesson does what agent-fundamentals M8's Lesson 7 already did once with the built agent: it closes the ecosystem map, now that the operations layer is in hand and not just a promise in DISEÑO.md.

You're going to see, run for real, the exact limit of what this guide's gate can and can't judge — and, from there, a tour through the neighboring guides that exist specifically to carry this same operated agent beyond that boundary: infrastructure and incidents, the response's semantic quality, genuine cost reduction, generic resilience in depth, and security against manipulation.

Connection to the module

This lesson adds no new discipline to the operations layer — it exists to be explicit about the disciplines this guide, by design, never covered, and why that absence is a boundary drawn from DISEÑO.md, not an oversight. Lesson 8 closes the capstone with the "production-ready" checklist that assumes, precisely, everything this lesson traces.

Analogy: the restaurant survived the year, but it's still a single location

This module's introduction's restaurant survived a complete year of real service: the daily inspection, the accountant, the stopwatch, the backup generator, the chef-substitution protocol — the five operations pieces, working together. But a restaurant that survived a year, with a single kitchen, still isn't a chain with a municipal health inspection over the whole building (infrastructure and incidents, not just the kitchen). It still doesn't have a food critic confirming, dish by dish, that the food tastes good — not just that it reached the table with the right dish and on time (semantic quality). It still hasn't negotiated with its suppliers to lower the cost of the ingredients the accountant already knows how to measure (reducing cost, not just measuring it). It still hasn't passed the complete certification from a fire department auditing every emergency system in the entire building, not just the kitchen's generator (generic resilience, in depth). And it still doesn't have a guard at the door confirming whoever enters isn't there to manipulate the staff into breaking protocol (security). None of those five things is needed for the restaurant to serve well this year — but all of them are needed for it to grow beyond it.

Worked example: the gate's exact limit — it passes form, it never judges tone

Before naming the neighboring guides, it's worth seeing, run for real, this entire capstone's most concrete and least debatable limit: Lesson 5's gate never reads the final text the agent shows the user — only the tool it chose, its result's shape, the exact value, the cost, and the latency.

import harness as hn

# Mismo guion que book_focus_pro_3h_ana -- misma secuencia de tools,
# mismo booking_id, PERO con una respuesta final deliberadamente cortante.
rude_script = [
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "list_rooms", "input": {}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_02", "name": "get_quote",
         "input": {"room": "Focus", "tier": "pro", "hours": 3}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_03", "name": "book_room",
         "input": {"room": "Focus", "tier": "pro", "hours": 3, "member": "Ana"}}]},
    {"stop_reason": "end_turn", "content": [
        {"type": "text", "text": "Ya. Dale."}]},
]

report = hn.run_regression_gate(hn.CASE_SET, overrides={"book_focus_pro_3h_ana": rude_script})
hn.print_gate_summary(report)

What to expect:

=== GATE: PASS (5/5) ===
  quote_focus_pro_3h                     PASS
  quote_focus_basic_3h                   PASS
  book_focus_pro_3h_ana                  PASS
  book_boardroom_pro_1h_sofia            PASS
  book_and_cancel_studio_basic_1h_diego  PASS

PASS (5/5) — the gate noticed absolutely nothing different. And it's completely right not to notice: run_case (M5) never reads final["content"][0]["text"] — it only inspects the chosen tool, the result's schema, book_room's exact value, the cost, and the latency, none of which changed. "Ya. Dale." is, to this gate's four deterministic comparisons, just as valid a response as "Reservé Focus pro 3h para Ana. Confirmación #1." — because neither of the two questions this gate knows how to answer is "does the response sound professional?" If Lesson 5's v2 had introduced this exact tone change instead of skipping the quote, this guide's gate would have let it through with no alarm at all — not from a design oversight, but because that question, quite deliberately, was never within its scope.

The five neighboring guides

This guide operates the agent agent-fundamentals built. Around it there are five more sister guides, each one taking that same operated agent further in a direction this guide, by design, never covered:

agents-in-production  ← THIS GUIDE (operate: observe, measure, gate, harden+version)
   │
   ├─► sre-and-incident-response
   │     Infrastructure, not the application: SLI/SLO for a Lambda and an
   │     API, Prometheus/Grafana/CloudWatch, an incident's complete
   │     lifecycle with roles and severities, blameless postmortems. This
   │     guide operates ONE agent run; that guide operates the complete
   │     infrastructure sustaining thousands of runs at once.
   │
   ├─► evaluation-frameworks
   │     The response's SEMANTIC quality: trajectory evaluation,
   │     JUDGED tool-call accuracy, LLM-as-judge, golden datasets,
   │     evaluation CI/CD. This lesson's worked example made
   │     exactly the gap that guide fills visible.
   │
   ├─► cost-optimization-caching
   │     REDUCING cost: prompt caching, cost-based model selection,
   │     batching, tiered optimization. This guide measures cost
   │     (M3) as one more signal -- it never teaches how to lower it.
   │
   ├─► resilience-and-reliability-patterns
   │     Generic resilience patterns, in depth: bulkheads,
   │     graceful degradation, load shedding, real random
   │     backoff+jitter, measured with thread exhaustion and retry
   │     storms, for any HTTP dependency -- not just an agent's
   │     tool-call layer, which this guide's M6 hardened.
   │
   └─► agent-security-and-sandboxing
         Hardening the agent against malicious users: prompt
         injection, least privilege, sandboxing, human-in-the-loop,
         guardrails, secrets. This guide assumes the agent is
         already hardened against infrastructure failures -- never
         against manipulation.

And, as agent-fundamentals M8 (Lesson 7) already traced, the agent's components — long-term memory, RAG, context engineering, MCP, multi-agent orchestration — keep living in their own sister guides. This guide operates them as part of a run when they exist (measures them, logs them) — it never builds them or goes deep into them.

Two boundaries worth keeping in mind, because they're the easiest to confuse

  • Cost: measuring (this guide) versus reducing (cost-optimization-caching-guide). This guide's M3 stops at "how much did this run cost and why" — never at "how do I lower it." If the question changes to "should we cache this system prompt?" or "would a smaller model solve this case just as well and cheaper?", that question belongs, unambiguously, to the sister guide.
  • Resilience: an agent's tool-call layer (this guide's M6) versus generic distributed systems (resilience-and-reliability-patterns-guide). M6's CircuitBreaker protects a specific agent tool, with modeled, deterministic backoff. That guide builds the same vocabulary — cited, never repeated — at a completely different scale: any HTTP dependency in a distributed system, with real random jitter and measured against thousands of simulated clients.

How to use this map

You don't need to go through all five neighboring guides for this capstone to have kept its promise: you already know how to instrument, measure, gate, and harden an agent that already works. The neighboring guides are there for when operating your agent genuinely runs into the specific limit each one solves:

  • Do you need to alert on a service's error rate, or write a blameless postmortem? → SRE and incidents, without exception.
  • Is the real question whether the agent's response is clear, correct, and well reasoned — not just whether its form didn't break? → evaluation frameworks.
  • Is the cost M3 measures already a real budget concern, and you need to lower it? → cost optimization & caching.
  • Do you need bulkheads, graceful degradation, or load shedding for a complete distributed system, not just one agent tool? → resilience & reliability patterns.
  • Are you going to expose the agent to real users or untrusted inputs? → agent security & sandboxing, without exception.

The same rule holding up every module in this guide — add a capability only when the task needs it, never "just in case" — applies at this complete-ecosystem scale too.

Common mistakes

  1. Thinking "the gate passed" means "the agent's response is good." This lesson's worked example disproves that with run evidence: a curt, unprofessional text passes the gate exactly the same as a careful one, because the gate never inspects the text. "Passed the gate" means, precisely, "the form didn't break" — never "the quality is acceptable."

  2. Confusing cost/latency threshold validation (M5) with real cost optimization. check_cost_threshold confirms a run stayed under a known budget — it doesn't lower that budget. Lowering the real cost (caching, model selection) is, without exception, cost-optimization-caching-guide's job.

  3. Thinking M6's CircuitBreaker "already is" complete distributed-system resilience. It protects one agent tool, with deterministic data — it doesn't build bulkheads, doesn't measure thread-pool exhaustion, doesn't use real random jitter. That complete depth lives, entirely, in resilience-and-reliability-patterns-guide.

  4. Jumping straight to agent-security-and-sandboxing or evaluation-frameworks without having first operated an agent with this guide. Both rest on a solid understanding of what observing, measuring, and gating a run means — without that foundation, those guides' vocabulary feels like an isolated layer, instead of a reasoned extension of what you already built.

  5. Adopting all five neighboring guides at once, "to be safe." The same discipline holding up every module in this guide — one capability at a time, verified before moving on — applies to the complete ecosystem too. Adopting SRE, semantic evaluation, cost reduction, generic resilience, and security simultaneously, without having felt the concrete need for each one, multiplies complexity without necessarily multiplying value.

Exercises

Exercise 1: Which guide solves it? (Easy)

For every need, name the right neighboring guide: (a) "I need to know if 99% of my Reservo runs respond in under 2 seconds, measured with production's real clock"; (b) "I need a model to evaluate whether the price explanation the agent gave the customer is clear and correct"; (c) "monthly token spend is already significant, I want to cache the system prompt"; (d) "I need to defend the agent from a user hiding instructions inside a fake booking."

See solution

(a) sre-and-incident-response-guide — infrastructure latency, measured with the real clock, with SLI/SLO — not this guide's M4 modeled latency.

(b) evaluation-frameworks-guide — semantic judgment on a response's quality, exactly the gap this lesson's worked example made visible.

(c) cost-optimization-caching-guide — reducing cost with prompt caching, not just measuring it like this guide's M3 does.

(d) agent-security-and-sandboxing-guide — defense against prompt injection, a type of manipulation no form validation in this guide covers.

Exercise 2: This guide or a neighbor (Medium)

Contrast each capability this capstone DOES have with the related neighboring capability it DOESN'T have: (a) M6's CircuitBreaker vs. bulkheads and load shedding; (b) check_cost_threshold (M5) vs. reducing real cost; (c) rollout_decision (M7, form comparison) vs. a semantic judge comparing two versions.

See solution

(a) M6's CircuitBreaker (this capstone) protects one specific tool of an agent, with state persisting across runs, over deterministic data. Bulkheads and load shedding (resilience-and-reliability-patterns-guide) operate at the complete distributed system level — isolating resources across different dependencies, deliberately shedding traffic when the whole system is overloaded — a much broader scope than a single tool.

(b) check_cost_threshold (this capstone) confirms a run stayed under an already-fixed budget — a comparison, not an optimization. Reducing real cost (cost-optimization-caching-guide) changes the budget itself: caching a repeated prompt, choosing a cheaper model for a simple task, batching calls together.

(c) rollout_decision (this capstone) compares, with ==, whether v2 still chooses the same tool and the same exact value as v1 on every CASE_SET case — a literal comparison. A semantic judge (evaluation-frameworks-guide) would compare whether v2's response quality is equal, better, or worse than v1's — a question no == can answer, and one this guide, quite deliberately, never tries to answer.

Exercise 3: Trace Reservo's path toward enterprise-grade operation (Hard)

This capstone's Reservo agent is going to handle thousands of real bookings per day, with an on-call team rotating shifts, a token budget already worrying finance, and the requirement that responses sound professional without exception. Order which guides you'd go through, in what sequence, and justify the order.

See solution

A reasonable order, from the foundation outward:

  1. agents-in-production-guide, complete (M1-M8). You already have it: the agent instrumented, measured, gated, and hardened, with a safe rollout. Without this, none of the following guides have anything to rest on.

  2. sre-and-incident-response-guide, for the on-call team and infrastructure. "Thousands of bookings per day" and "a team rotating shifts" are, precisely, the language of SLI/SLO, incident roles, and postmortems — the infrastructure layer sustaining the volume this capstone never needed to simulate.

  3. evaluation-frameworks-guide, for "responses sounding professional without exception." This lesson's worked example showed, with evidence, that this guide's gate never certifies tone. With real volume, that certification stops being optional.

  4. cost-optimization-caching-guide, for the budget already worrying finance. This guide's M3 already measures cost precisely — the next step, once the number genuinely worries someone, is lowering it.

Why that order: operate first, then support infrastructure for the volume, then certify the quality the volume exposes, then optimize the cost the volume inflates — every guide rests on the previous one already being solved. agent-security-and-sandboxing and resilience-and-reliability-patterns don't appear on this list because the described scenario doesn't mention malicious users or external HTTP dependencies beyond the tools M6 already hardened — adding them "just in case" would repeat, again, the mistake the "How to use this map" section warned against.

Summary and next step

  • We ran, with real evidence, this entire capstone's most concrete limit: Lesson 5's gate passes 5/5 even with a curt, unprofessional final response, because it never inspects the text — only the tool, the schema, the value, the cost, and the latency.
  • We toured the operations ecosystem's five neighboring guides: SRE and incidents (infrastructure, not the application), evaluation frameworks (semantic quality, the exact gap the worked example made visible), cost optimization & caching (reducing, not just measuring), resilience & reliability patterns (generic resilience in depth, beyond a single tool), and agent security & sandboxing (hardening against manipulation, not just infrastructure failures).
  • The adoption rule stands: this complete guide is enough for the scope it promised; every neighboring guide gets added only when your operated agent genuinely runs into the specific limit that guide solves.

Next lesson: 08 — Project: Ship the Production-Ready Agent. We close the capstone and the entire guide with the final checklist and the four artifacts generated end to end in a single run.

Additional resources

  1. Anthropic — Building effective agents — On when to add operational complexity (more disciplines, more guardrails) and when not to — the same question this lesson applies at ecosystem scale.
  2. Anthropic — Tool use overview — The protocol sustaining both this guide and its five operations neighbors.
  3. evaluation-frameworks-guide — that guide's complete index, for when the gap this lesson's worked example showed becomes a real need.
  4. Python — Module scope and lifecycle — The same technical foundation underlying, throughout this guide, why BOOKINGS's or a CircuitBreaker's state lives and dies with the process hosting it.
← PreviousThe Resilience LayerNext →Project: Ship the Production-Ready Agent