Module 1: Why Operating Is Different From Building

Module 1: Why Operating Is Different From Building

Next →

Description

agent-fundamentals-and-tool-calling-guide ended with an agent that works: the Reservo agent, able to list rooms, quote, book, and cancel, chaining those four tools across several steps, self-correcting when the model requests something invalid, and answering with a confirmation grounded in real results. That agent lives in reservo_agent.py, behind a single function: run_reservo_agent(question, model_script). You give it a question and a turn script, and it gives you back a text answer.

This guide picks up exactly there — without rebuilding a single line of that function — and asks a different question: what changes when that same agent stops solving the test script you wrote yourself, and starts solving whatever real users ask it, one after another, over an entire day? The short answer, and the theme of this module's eight lessons: almost everything you need to trust a system in production is information the agent, as it was built, does not give you. And giving it to you — without touching its logic — is a craft of its own, distinct from the craft of building it. That craft is operating.

Hard rule for this guide (read before continuing)

This guide inherits, literally, the hard rule of agent-fundamentals-and-tool-calling-guide: the model's decision is never executed. When a lesson says "the model (claude-sonnet-5, concept) decided to call get_quote," that's a hand-written turn script — never a real API call. That part doesn't change.

What this guide adds is a layer of its own rules, because the topic now is measuring a system, not just building it, and measuring wrong is worse than not measuring:

  • Cost is estimated, and the cost arithmetic actually DOES run. The text-to-tokens conversion uses the same honest convention from agent-fundamentals: len(text) // 4, always labeled as an order-of-magnitude estimation — never as the exact count of a real tokenizer. The pricing of claude-sonnet-5 is a fixed, cited constant: $3.00 per million input tokens, $15.00 per million output tokens — the list price, verified against the official Claude documentation. (A launch promotional price, $2.00/$10.00, existed, valid only through August 2026; this guide uses the list price because it's the number that stays true after that date.) The cents calculation using that estimate and that fixed price really does run, and you'll see its real output starting in lesson 05.
  • Latency is modeled, never measured with a real clock. In no code block of this guide will you see time.time() or time.perf_counter(). Instead, each tool has a fixed latency, declared in a dictionary (TOOL_LATENCY_MS), and a run's latency is the sum of its steps' latencies. This is a deliberate simplification: in real production, latency is measured with a stopwatch around each call — that's trivial to code and teaches nothing new. What does teach something is deciding what to do with that measurement once you have it: which tool dominates the total, how stable it is from one run to the next, when a figure is suspicious. Modeling latency with fixed data makes every example byte-for-byte reproducible, so you can confirm exactly what you see here on your own machine.
  • No randomness and no real clocks in the data. No random, no datetime.now(), no uuid4(). Any identifier that needs to be unique — like a trace_id, which you'll build in depth in Module 2 — is deterministic: a counter, or a hash of the run's inputs.
  • Operations engineering really does run, for real, with Python 3.14.0 and its standard library (logging, json, dataclasses, statistics, among others depending on the module). No network, no git, no gh. Everything runs locally, and every output you see in a "What to expect" block is the real output of having run that code.

Hold onto this sentence, because you'll use it in every module that follows: the model decides (concept); the agent acts (already built, in agent-fundamentals); this guide observes, measures, gates, and hardens that acting (real engineering, here).

Where we are in the ecosystem

Agents in production — operating the Reservo agent
├── Module 1: Why Operating Is Different From Building  ← YOU ARE HERE
│   → The bridge from agent-fundamentals, an uninstrumented run,
│     the signals that matter, the operating-vs-building boundary
├── Module 2: Structured Logging and Tracing a Run
├── Module 3: Measuring Cost and Tokens per Run
├── Module 4: Measuring Latency Honestly
├── Module 5: Regression Evals as a Production Gate
├── Module 6: Failures at Scale — Backoff, Circuit Breakers, and Rate Limits
├── Module 7: Versioning and Safe Rollout
└── Module 8: Project — the Reservo Agent in Production

This is Module 1 of 8. It doesn't add a single new tool, nor a single new line to the agent's loop — that's already solved, and solved well, in agent-fundamentals-and-tool-calling-guide. What it does is lay out, with precision, the problem the following seven modules solve one by one: an agent that works is not the same as an agent you can trust to operate. Modules 2 through 7 are the four disciplines that close that gap — observe, measure, gate, harden, and version — and Module 8 brings all of them together on the same Reservo agent, with real evidence: four deliverable files (RUN_LOG.jsonl, a metrics summary, a regression_report.json, an AGENT_CHANGELOG.md).

The agent, uninstrumented: the same question as always

Before naming any discipline, it's worth seeing the problem with your own eyes. Reservo, as it stood at the end of agent-fundamentals M8, solves that guide's canonical task: "book Focus pro 3h for Ana." The turn script (concept, claude-sonnet-5) explores the rooms, trips over an invalid tier, self-corrects, books, and answers:

import reservo_agent as ra

model_script_demo = [
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "list_rooms", "input": {}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_02", "name": "get_quote",
         "input": {"room": "Focus", "tier": "premium", "hours": 3}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_03", "name": "get_quote",
         "input": {"room": "Focus", "tier": "pro", "hours": 3}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_04", "name": "book_room",
         "input": {"room": "Focus", "tier": "pro", "hours": 3, "member": "Ana"}}]},
    {"stop_reason": "end_turn", "content": [
        {"type": "text", "text": "Reservé Focus pro por 3 horas para Ana. Total $60.00. Confirmación #1."}]},
]

final, history = ra.run_reservo_agent("Reserva Focus pro 3h para Ana", model_script_demo)
print("RESPUESTA:", final["content"][0]["text"])

What to expect:

RESPUESTA: Reservé Focus pro por 3 horas para Ana. Total $60.00. Confirmación #1.

One line. That print is, literally, everything a system that calls run_reservo_agent and uses its result needs to answer the user — and it's exactly what most real agent integrations do: they receive the question, call the function, take final["content"][0]["text"], hand it back to whoever asked. The task got solved. Nobody complained. If this were all that existed, there'd be nothing to operate.

But now ask yourself these five questions, with just that one line in front of you:

  1. How many steps did the agent take to reach that answer? One? Ten?
  2. Which tools did it call, and in what order?
  3. Did anything fail along the way? Did the model request something invalid and have to correct itself?
  4. How much did this run cost? How many input and output tokens did it consume, and how many cents does that equal?
  5. How long did it take? How much of the time did each tool eat up?

With the code above, you can't answer a single one of the five. history does exist as a local variable while the process is still alive — you could, at this very moment, inspect it by hand — but the instant the function returns and nobody else looks at it, that information is lost. And even inspecting it by hand, you still can't reach questions 4 and 5: there's no token count, no cost, no time anywhere in that structure. history records what happened, not how much it cost for it to happen nor how long it took to happen. Lesson 03 of this module stops exactly at this point, in more detail, putting the rest of the five questions to the test one by one.

The four disciplines of this guide

agent-fundamentals answered "how do I build an agent that solves a multi-step task?" This guide answers a different, later question: "how do I trust that this same agent, running while I'm not watching it, keeps doing the right thing?" That trust is built in four layers, each resting on the one before:

1. OBSERVE    -> what exactly happened in this run?              (Module 2)
   Structured logging + a trace_id that correlates every
   step of the loop, end to end.

2. MEASURE    -> how much did it cost and how long did it take?  (Modules 3-4)
   Cost per run (estimated tokens x fixed price) and latency
   per tool and total (modeled, honest about its limit).

3. GATE       -> is it still behaving as expected?                (Module 5)
   A deterministic regression harness -- of SHAPE, never a
   semantic judge -- that fails the build if something broke.

4. HARDEN
   AND VERSION -> does it survive repeated failures, and can I
                  change it without betting blind?                (Modules 6-7)
   A per-tool circuit breaker between runs, and a disciplined
   comparison between an old version and a new one before
   deciding GO/NO-GO.

Notice the order: you can't gate (layer 3) what you can't measure (layer 2), and you can't measure meaningfully what you can't observe first (layer 1). That's why Module 2 — logging and tracing — is the first of the seven remaining: it's the foundation the other six stand on. Module 8 doesn't add a fifth discipline — it brings the four together on the same Reservo agent and delivers the evidence.

This module, Module 1, doesn't build any of the four yet. It introduces logging and a minimal trace of a run in lesson 08 — enough for you to see, with your own hands, the general shape of "wrapping" a run with instrumentation — but the in-depth development of that layer (a trace_id that truly correlates, the complete fields of each event, a RUN_LOG.jsonl file) is Module 2's work. What this module does do, fully, is lay out the problem with precision and name the four operational signals you'll measure starting in lesson 04: error rate, per-tool failure rate, cost per run, latency per run.

The case that keeps accompanying the guide: Reservo, without rebuilding anything

The system is the same as always — Reservo, coworking room bookings — and the four tools are exactly the ones agent-fundamentals M2 declared and M8 assembled, without changing a line:

  • list_rooms() — lists the rooms with their base hourly rate, in cents. Read-only.
  • get_quote(room, tier, hours) — quotes a room. Returns {price_cents}. Read-only.
  • book_room(room, tier, hours, member) — creates a booking. Returns {booking_id, confirmed}. Write.
  • cancel_booking(id) — cancels a booking. Returns {cancelled}. Destructive.

The two price anchors you already know remain the reference point:

Focus basic 3h  ->  2500 * 3            = 7500  ($75.00)
Focus pro   3h  ->  2500 * 3 * 80 // 100 = 6000  ($60.00)

And the runner that orchestrates all of this — the one we called above, run_reservo_agent — is literally the one from agent-fundamentals M8, lesson 05: it validates every argument before executing, catches any real exception, retries what's transient, cuts off what takes too long, and never lets an uncontrolled exception propagate to its caller — except RuntimeError when the iteration cap is exhausted, the one case that does propagate. This guide imports that file, calls it, and builds around it — it never rewrites its internal logic. When a lesson says "the runner retries this," it's describing behavior that already exists and has already been tested, not something being invented here.

What's genuinely new in this guide are the artifacts of the operating layer — modules built around the agent, never inside it: a structured logger, a cost calculator, a latency model, a regression harness, a per-tool circuit breaker, and a versioned configuration record. Identifiers and code, always in English; prose and comments, in Spanish; money, always in int cents.

Prerequisites

Required knowledge:

  • ✅ Having completed (or knowing well) agent-fundamentals-and-tool-calling-guide, especially Modules 4 (the loop), 7 (robustness), and 8 (the complete Reservo agent). This guide assumes that agent is already built and doesn't re-explain tool_use/tool_result, a tool's contract, or is_error.
  • ✅ Python: functions, dictionaries, list comprehensions, try/except. All the engineering in this guide is stdlib.

Recommended:

  • ✅ Having felt, at some point, the question "why did this run cost so much?" in front of a system that only gives you a response and nothing else — that frustration is exactly the problem this module solves.

NOT required:

  • ❌ You don't need an API key or an internet connection: the model's decision is still concept, and all the operations engineering runs 100% locally.
  • ❌ You don't need to know an observability framework (Datadog, LangSmith, Sentry). The patterns — structured logging, trace_id, regression harness, circuit breaker — are the same with any of them; here you build them by hand to understand them.
  • ❌ You don't need to know about infrastructure, SRE, Prometheus, or Grafana: that's sre-and-incident-response-guide, a neighboring guide that operates infrastructure, not the agent itself. Lesson 06 of this module draws that boundary with precision.

Environment:

  • ✅ Python 3.14.0 with its standard library. Nothing to install.
  • ✅ A text editor and a terminal. That's it.

Module roadmap

Lesson 01 — Module introduction (this one)

The bridge from agent-fundamentals, this guide's hard rule, an uninstrumented run and the five questions it can't answer, the four disciplines, and the map of the eight lessons.

Lesson 02 — The agent already works in a notebook, now what?

What exactly changes when the same run_reservo_agent stops solving a fixed test script and starts solving different questions, one after another, that nobody wrote in advance.

Lesson 03 — What you cannot see without instrumentation

The module's central analogy — a car without a dashboard — put to the test with real code: which questions can be answered by looking at history by hand, and which ones aren't even there.

Lesson 04 — The operational signals that matter

Four signals, precisely defined and calculated over real runs: error rate (at the run level), per-tool failure rate (at the tool-call level), cost per run, latency per run.

Lesson 05 — A first look at cost, latency, and errors

The first time this guide actually calculates how much a run cost and how long it took — with the fixed price formula and the modeled latency, run over the canonical run.

Lesson 06 — Operating vs. building: the boundary

Exactly where what agent-fundamentals already solved (the loop, is_error within a run) ends and this guide begins; and where this guide (the agent) ends and sre-and-incident-response-guide (the infrastructure) begins.

Lesson 07 — The brief: run the Reservo agent for real users

This guide's case, laid out as a concrete brief: four business questions that can't be answered today, and a map of which module answers which.

Lesson 08 — Mini-project: wrap a run and see inside

You build your first instrumentation wrapper — lightweight, without formal logging yet — around run_reservo_agent: cost in cents, modeled latency, and lesson 04's signals, all together, over a real batch of runs.

Progression map

Lesson 01 (this)  → The bridge, the hard rule, the four disciplines
Lesson 02         → What changes with real users
Lesson 03         → The car without a dashboard, with real code
Lesson 04         → The four signals, defined and calculated
Lesson 05         → Cost + latency + errors, for the first time
Lesson 06         → The operating-vs-building boundary (and vs. infra)
Lesson 07         → This guide's brief
Lesson 08         → Mini-project: the first wrapper

Difficulty: ⭐ ──────────────────▶ ⭐⭐

What you'll achieve in this module

By completing the 8 lessons, you'll be able to:

  1. Explain why "it works in the notebook" is not the same as "it's ready to operate," with the concrete example of Reservo.
  2. Name the five questions an uninstrumented run can't answer, and why none of the five live in history.
  3. Distinguish the four disciplines of this guide — observe, measure, gate, harden+version — and which module builds each one.
  4. Precisely define error rate, per-tool failure rate, cost per run, and latency per run — and calculate the first two over real runs.
  5. Trace the boundary between what agent-fundamentals already built and what this guide operates, and between what this guide operates (the agent) and what sre-and-incident-response-guide operates (the infrastructure).
  6. Wrap a run of run_reservo_agent with a minimal measurement of cost, latency, and signals — your first operations artifact, executed.

Before and after

BEFORE the module:
→ "If the agent answers well, it's already ready for production"
→ "history already tells me everything I need to know about a run"
→ "measuring cost is a billing detail, not engineering"
→ "latency is measured by putting a stopwatch on it, I already know how"

AFTER the module:
→ An agent that solves a script's task is not the same as
  one you can trust with real users, at scale
→ history records WHAT happened, never HOW MUCH it cost or
  HOW LONG it took
→ cost and latency per run are first-class operational signals,
  with the same seriousness as error rate
→ observe, measure, gate, and harden are FOUR distinct
  disciplines, each resting on the one before

Traps to avoid in this module

1. "This module is going to rebuild the agent, better this time"

No. Not a single line of reservo_tools.py, reservo_contracts.py, reservo_robust.py, or reservo_agent.py changes in this guide. If any of that logic looks improvable to you, that's a note for a review of agent-fundamentals, not a silent rewrite here — rewriting it would break traceability of what was tested where.

2. "If a tool call's is_error is already handled in M7, there's nothing else to harden"

There's a real distinction that lesson 06 traces with precision: agent-fundamentals M7 solves "what does the agent do when THIS call to THIS tool fails, right now, within this run?" (self-correction, a bounded retry). This guide, in Module 6, solves a different question: "what does the system do when that tool has been failing for several runs in a row?" — a circuit breaker with state that persists between runs, not within a single one.

3. "Measuring latency is just putting time.perf_counter() around a call"

Programmatically, yes — and that's exactly why this guide doesn't stop there. What actually demands real work is deciding what to do with that measurement: which tool dominates the total, which percentile matters, when a figure is a signal versus noise. That's why this guide models latency with fixed data: to focus on those questions without the example's reproducibility depending on how long your machine happened to take at that exact moment.

4. "The cost of a run is a billing problem, not an engineering one"

Not in this guide. Measuring how much a run costs — and why it cost that — is a first-class operational signal, just as important as whether it failed or not. What this guide does not do is teach you to reduce that cost (prompt caching, model selection, batching) — that's cost-optimization-caching-guide, named precisely where it belongs.

5. "This is already SRE, or already semantic quality evaluation"

No. This guide operates the agent — its runs, its tool calls, its tokens, its prompts — at the application level, with pure Python, zero cost. Infrastructure SRE (a service's SLI/SLO, Prometheus, an incident's lifecycle) is sre-and-incident-response-guide. And this guide's regression gate (Module 5) never judges whether a response is semantically good — that's evaluation-frameworks-guide — it only confirms that the shape didn't break: the schema, the tool chosen, cost and latency under a fixed threshold.

How to work through this module

  1. Run every example yourself. Every lesson comes with runnable code and its real "What to expect." Seeing 185 milliseconds or 0 cents come out of your own terminal is worth more than reading it.
  2. Don't confuse "modeled" with "made up." When a lesson says latency is modeled, it doesn't mean the number is arbitrary — it means it's a fixed, declared value, not a clock measurement. Being honest about that difference is part of what this module teaches.
  3. The mini-project (lesson 08) is the synthesis. There you'll wrap run_reservo_agent with your first real measurement of cost, latency, and signals — the conceptual starting point for Modules 2 through 8.

Estimated time:

Lesson 01 (this)  →  20 min reading
Lesson 02         →  20 min + running the example
Lesson 03         →  25 min + running the example
Lesson 04         →  25 min + running the example
Lesson 05         →  25 min + running the example
Lesson 06         →  20 min reading
Lesson 07         →  20 min reading
Lesson 08         →  35 min + building the wrapper

Total: ~3 hours

Evidence of success

Before moving on to Module 2 (Structured Logging and Tracing a Run), you should be able to:

  • ✅ Explain, with the Reservo example, why "the agent answers well" is not the same as "the agent is ready to operate."
  • ✅ Name the five questions an uninstrumented run leaves unanswered.
  • ✅ Distinguish this guide's four disciplines and their dependency order.
  • ✅ Calculate, over a real run, its per-tool failure rate and say whether the run completed or not.
  • ✅ Estimate, with the fixed formula, the cost in cents and the modeled latency of a run.
  • ✅ Trace the boundary between this guide, agent-fundamentals, and sre-and-incident-response.

Summary

  • This guide operates the Reservo agent that agent-fundamentals-and-tool-calling-guide already built — it doesn't rebuild it, it wraps it.
  • Hard rule: the model's decision is still concept (claude-sonnet-5); the operations engineering — logging, cost, modeled latency, regression, circuit breaker, versioning — actually runs with Python 3.14.0.
  • We demonstrated it running: given "book Focus pro 3h for Ana," the agent answers with a single line of text — and that line, by itself, answers not one of five basic operational questions: steps, tools, failures, cost, latency.
  • This guide's four disciplines are observe → measure → gate → harden+version, each resting on the one before, developed in Modules 2 through 7 and joined in Module 8.
  • The boundary is double: toward agent-fundamentals (the loop and basic error handling are already built, not repeated here) and toward sre-and-incident-response (that guide operates infrastructure; this one operates the agent).

Next lesson: 02 — The Agent Already Works in a Notebook, Now What?. We see, with executed code, exactly what changes when run_reservo_agent stops solving the script you wrote and starts solving questions you don't control.

Additional resources

  1. Anthropic — Tool use (function calling) overview — The complete protocol the Reservo agent already implements, and that this guide operates without re-explaining.
  2. Anthropic — Building effective agents — On why an agent's reliability in production depends on the discipline of operating it, not just building it well once.
  3. Anthropic — Pricing — The source of claude-sonnet-5's list price, which this guide fixes as a constant starting in lesson 05.
  4. Python 3.14 — What's New — The exact version all the engineering in this guide runs on.
  5. Python — logging — The library Module 2 develops in depth, and this lesson introduces by name.
Next →The Agent Already Works in a Notebook, Now What?