Module 1: Why Operating Is Different From Building

The Brief: Run the Reservo Agent for Real Users

Next →

Description

Lessons 01 through 06 of this module did three things: showed the problem (an uninstrumented run is a black box), named the signals that matter (error, per-tool failure, cost, latency), and precisely traced where what's already built ends and what this guide operates begins. This lesson brings the three together in one place, in the shape this kind of work arrives in real life: not as a list of concepts, but as a brief, with concrete business questions someone — a product owner, a manager, Reservo's owner — needs you to answer before letting real users use the agent.

Connection to the module

This is the direct bridge into Modules 2 through 8. Each of this lesson's four brief questions has a specific module that answers it — and seeing them laid out as business questions, not module names, is what gives meaning to why this guide is organized in that order.

The brief

Imagine Reservo, the coworking room-booking system, decides to open its conversational agent to real users — no longer the dev team testing scripts, but anyone with an account, typing whatever occurs to them, at any hour. Before approving that, whoever runs the product asks you four questions. None is a technical whim — each is exactly the kind of question a real business asks before trusting real traffic to an automated system:

Question 1: "How much is this going to cost us to run per month?"

You don't know today. As lesson 05 confirmed, calculating the cost of one run is already possible — claude-sonnet-5's fixed formula plus the token estimation — but a monthly business number needs to sum the cost of all the runs over a period, broken down by volume and by task type. That's Module 3.

Question 2: "Is it going to respond fast, or is it going to feel slow to the customer?"

You don't know that yet either, beyond the individual run you measured in lesson 05. A business answer needs to know what latency percentile a typical customer experiences (not just the average, which can hide the customers with the worst experience) and which specific tool contributes the most when something feels slow. That's Module 4.

Question 3: "How do we know if something broke, without a customer complaining first?"

This is, at bottom, the question that drove this entire module: today, the only way to find out something failed is for someone to look at history by hand, at the exact moment it happens — and you already saw, in lesson 03, that not even that works when the run breaks down completely. Answering this question seriously needs two things: a record of every run that survives beyond the individual call (Module 2, structured logging and trace_id), and a gate that automatically confirms the agent's behavior is still what's expected before a change reaches production (Module 5).

Question 4: "What happens if we change the prompt or add a new tool, without breaking what already works?"

Changing the system — a new prompt, a new tool, a different model — is inevitable in any product that evolves. The business question isn't "can we change it?" — of course you can — but "how do we know, with evidence, that the change didn't break something that used to work?" That combines three pieces: the same regression gate from Question 3 (Module 5), a mechanism that stops insisting on a tool that started failing consistently (Module 6), and a versioned record that lets you compare the old version against the new before deciding whether the change stays (Module 7).

The brief's complete map

BUSINESS QUESTION                                  MODULE THAT ANSWERS IT
────────────────────────────────────────────────────────────────────────
"How much will it cost per month?"              ->  M3 -- cost per run, aggregated
"Will it respond fast?"                          ->  M4 -- latency, p50/p95
"How do we know if something broke?"             ->  M2 (observe) + M5 (gate)
"Can we change something without breaking it?"   ->  M5 + M6 (harden) + M7 (version)

EVERYTHING TOGETHER, OVER THE COMPLETE RESERVO
AGENT                                            ->  M8 -- the capstone

None of the four questions can be answered with a single isolated signal — each one needs, at minimum, a whole module's discipline. And all four together are, literally, Module 8's content: operating the same Reservo agent with all four layers on top, and delivering real evidence — four files — that every question has an answer.

Worked example: a morning of real traffic, with no prior script

To feel the difference between "a script I wrote to test something" and "what a real business receives," run two tasks that didn't appear in any earlier lesson of this guide — an inquiry that never books anything, and a booking from a new person and a new room:

import reservo_agent as ra

# Un cliente que solo quiere saber el precio, sin reservar nada todavía.
script_quote_only = [
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "get_quote",
         "input": {"room": "Boardroom", "tier": "basic", "hours": 2}}]},
    {"stop_reason": "end_turn", "content": [
        {"type": "text", "text": "Boardroom basic 2h cuesta $160.00."}]},
]

# Un cliente nuevo, con un nombre que no apareció en ninguna lección anterior.
script_marta = [
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "get_quote",
         "input": {"room": "Focus", "tier": "basic", "hours": 2}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_02", "name": "book_room",
         "input": {"room": "Focus", "tier": "basic", "hours": 2, "member": "Marta"}}]},
    {"stop_reason": "end_turn", "content": [
        {"type": "text", "text": "Reservé Focus basic por 2 horas para Marta. Total $50.00. Confirmación #1."}]},
]

final_1, _ = ra.run_reservo_agent("¿Cuánto cuesta Boardroom basic 2h?", script_quote_only)
print("Consulta  ->", final_1["content"][0]["text"])

final_2, _ = ra.run_reservo_agent("Reserva Focus basic 2h para Marta", script_marta)
print("Reserva   ->", final_2["content"][0]["text"])

What to expect:

Consulta  -> Boardroom basic 2h cuesta $160.00.
Reserva   -> Reservé Focus basic por 2 horas para Marta. Total $50.00. Confirmación #1.

Boardroom basic 2h: 8000 * 2 = 16000 cents, $160.00 — confirmed. Focus basic 2h: 2500 * 2 = 5000 cents, $50.00 — confirmed. Two tasks solved correctly, neither reused from any earlier lesson. This is, precisely, the kind of variety the brief's four questions assume as a starting point: nobody at Reservo is going to hand-write a turn script for every real customer — the script (concept, claude-sonnet-5) is produced by the model, over questions nobody on your team saw coming.

Why the brief can't be accepted yet

With what exists today — the agent-fundamentals agent, with no operating layer on top — the honest answer to the brief's four questions is "I don't know, and I have no way to find out without looking at every run by hand, one by one." That's not a flaw in the agent — it's exactly the state any system that solves multi-step tasks is in the day before it gets instrumented. The difference between a team that can accept this brief with confidence and one that can't isn't having a better-built agent — it's having this guide's four disciplines on top of the same agent. That is exactly what Modules 2 through 8 build, one at a time.

Common mistakes

  1. Treating the brief's four questions as if they were one. Each needs a different discipline — cost, latency, observability+gate, versioning+hardening — and mixing them up leads to building the wrong layer first. For example, it makes no sense to try to answer "what happens if we change the prompt?" (Question 4) before having a regression gate (Question 3, Module 5) to compare against.

  2. Thinking answering one out of the four is enough to "be ready." A real business needs all four — knowing the monthly cost without knowing whether the system still works after a change is a partial, insufficient answer.

  3. Trying to fully answer the whole brief in this module. This module doesn't fully answer any of the four questions — it precisely lays them out and builds the vocabulary (lesson 04's signals) that the following modules use to actually answer them. That is, precisely, a Module 1's job: framing the problem, not solving it.

  4. Writing a "real customer" script and thinking that already solves production's variety. This lesson's worked example used two new tasks, but they're still scripts you wrote by hand — production's genuine variety comes from thousands of questions no human on your team wrote in advance. This lesson demonstrates the kind of variety, it doesn't replace it.

  5. Jumping ahead to build the circuit breaker or versioning before having observability. The brief's order isn't arbitrary: you can't decide with evidence whether a change broke something (Question 4) without first being able to measure (Questions 1 and 2) or first being able to observe (part of Question 3). That dependency order is, literally, the order of Modules 2 through 7.

Exercises

Exercise 1: Assign each question to its module, without looking at the map (Easy)

Without looking back at the "The brief's complete map" section, write from memory which module (2 through 7) each of the brief's four questions corresponds to. Then compare with the map and correct whatever needs it.

See solution
  1. "How much will it cost per month?" → Module 3 (cost per run, aggregated over a batch).
  2. "Will it respond fast?" → Module 4 (latency, percentiles).
  3. "How do we know if something broke?" → Module 2 (observe, logging + trace_id) and Module 5 (gate, regression).
  4. "Can we change something without breaking it?" → Module 5 (the same gate), Module 6 (harden, circuit breaker), and Module 7 (version, GO/NO-GO comparison).

Explanation: if your answer put Question 3 only in Module 5, or only in Module 2, review why both are needed: without Module 2, there's no record of what happened in each run to run Module 5's gate against; without Module 5, Module 2's record only tells you what happened, never whether what happened was expected.

Exercise 2: Write the fifth question Reservo should ask (Medium)

The brief's four questions cover cost, latency, observability+gate, and versioning+hardening. Based on what you learned in lesson 06 about the boundary toward sre-and-incident-response-guide, write a reasonable fifth business question Reservo should ask — but that none of this guide's eight lessons answers, because it belongs to that neighboring guide.

See solution

A reasonable answer: "what happens if the service exposing the agent goes down entirely — if the load balancer can't spread traffic, or the database behind book_room stops responding?" No lesson in this guide solves this question because it isn't a question about the agent's behavior — cost, its steps' latency, its tool calls' failure rate — it's a question about the availability of the infrastructure exposing it: a service's SLI/SLO, an incident's lifecycle with roles and severities, postmortems. That is, precisely, lesson 06's boundary: sre-and-incident-response-guide.

Other valid answers: "how long does the service take to recover after a failed deployment?", "do we have alerts if the service's 5xx error rate suddenly spikes?" — all legitimate business questions, all outside this guide's scope for the same reason.

Exercise 3: Order the four questions by dependency, and justify the order (Hard)

The brief's four questions don't have the same order of urgency as they do of technical dependency. Order them by what needs to be solved first for the next one to make sense, and justify each step of the order with one sentence.

See solution

A reasonable order, from earliest dependency to latest:

  1. Observability (part of Question 3, Module 2). Without a record of what happened in each run, none of the other three questions have data to work with — there's no cost to sum, no latency to average, no behavior to compare against a gate.

  2. Cost and latency (Questions 1 and 2, Modules 3 and 4). Both depend directly on the observability from the previous step — they need every run's record to calculate their signals — but they're independent of each other: they can be built in either relative order.

  3. The regression gate (the rest of Question 3, Module 5). It needs signals to compare — cost, latency, failure rate — to be able to set a threshold and decide pass/fail; that's why it depends on steps 1 and 2, not the other way around.

  4. Hardening and versioning (Question 4, Modules 6 and 7). Deciding whether a prompt change or a new tool "broke something" requires, as a prerequisite, already having a gate to compare against (step 3) — you can't decide GO/NO-GO without an objective criterion that already exists.

Why that order: each layer consumes the previous one's output as its raw material — the regression gate can't evaluate anything without signals to measure, and the signals don't exist without a record of what happened. It's exactly the same dependency principle lesson 01 showed with the four disciplines (observe → measure → gate → harden+version), now applied to the four concrete business questions.

Summary and next step

  • We laid out this guide's complete case as a business brief: four concrete questions — monthly cost, response speed, how to detect a break, how to change something without breaking what works — that can't be answered today with the agent as it stands.
  • We ran two new tasks, not scripted in any earlier lesson, to feel the kind of variety a real business brings — and confirmed the agent solves them correctly, with zero code changes.
  • We mapped each of the four questions to the modules that answer it (M2-M7), and confirmed, with a dependency exercise, why that order isn't arbitrary: each layer needs the previous one as its raw material.

Next lesson: 08 — Mini-project: Wrap a Run and See Inside. You build your first instrumentation wrapper — cost, latency, and the four signals, all together — around run_reservo_agent, over a real batch of runs. It's the first answer, still minimal, to this lesson's brief.

Additional resources

  1. Anthropic — Building effective agents — On the gap between a prototype that works and a system a business can trust with real traffic.
  2. Anthropic — Tool use (function calling) overview — The protocol this lesson's two new tasks run on, with no change at all.
  3. Anthropic — Pricing — The pricing reference that will back the real answer to the brief's Question 1, starting in Module 3.
  4. Python 3.14 — What's New — The version this lesson's two new tasks ran on.
← PreviousOperating vs. Building: The BoundaryNext →Mini-Project: Wrap a Run and See Inside