Why Version Prompts and Tools

Description

Module 6 left the Reservo agent protected against a tool that fails repeatedly: the circuit breaker opens, the agent stops insisting, and the system degrades under control. But there's a question that circuit breaker never answers, because it isn't designed to: what happens when the tool doesn't fail, and the problem is the agent decided to call the wrong tool?

This lesson stops, without writing the complete gate yet, on that danger's exact mechanism: how a single line added to the Reservo agent's system prompt can change which tool it decides to call for a question that, until yesterday, it handled well. It isn't a code bug — no function changes — it's a change in behavior, and an LLM-based agent's behavior can't be read with a text diff to know whether it's safe.

Connection to the module

This lesson is the whole module's motivation: before building a registry (Lesson 03) or running a gate (Lesson 04), you need to understand what you're trying to prevent. The rest of the module builds the machinery; this lesson explains why that machinery is needed.

Analogy: the recipe that changes one ingredient

A restaurant has a written recipe for its most popular dish. One day, the chef decides to "improve" the recipe by adding an instruction: "if the customer seems to be in a hurry, serve the dish without waiting for final table confirmation." The intent is good — less waiting for the customer. The problem shows up the first time a customer asked for just the menu to look at it, having ordered nothing yet, and the waiter, following the new instruction to the letter, brings them a dish nobody ordered.

Nobody made a cooking mistake. The dish is well made, with the right ingredients, served on time. The mistake is in when it was decided to act — the same kind of mistake that shows up when a system prompt tells the agent "be proactive" without being precise about that proactivity's limits.

The experiment: the same question, two prompts

You're going to see, without building any formal gate yet, the decision the Reservo agent would make under two versions of its system prompt, facing the exact same question.

SYSTEM_PROMPT_V1 = (
    "Eres el asistente de reservas de Reservo, un sistema de coworking. "
    "Ayudas a los usuarios a consultar salas, cotizar precios, reservar y "
    "cancelar reservas. Usa siempre las tools disponibles para cotizar y "
    "reservar -- nunca inventes un precio de memoria. Cuando el usuario "
    "solo pregunta cuanto cuesta algo, usa get_quote y NO reserves. Usa "
    "book_room unicamente cuando el usuario pide reservar de forma "
    "explicita."
)

SYSTEM_PROMPT_V2 = (
    "Eres el asistente de reservas de Reservo, un sistema de coworking. "
    "Ayudas a los usuarios a consultar salas, cotizar precios, reservar y "
    "cancelar reservas. Usa siempre las tools disponibles para cotizar y "
    "reservar -- nunca inventes un precio de memoria. Se proactivo: si ya "
    "tienes toda la informacion para completar una reserva, complétala "
    "directamente en vez de solo cotizar, para ahorrarle un paso al "
    "usuario. Usa book_room unicamente cuando el usuario pide reservar de "
    "forma explicita."
)

question = "Cuanto cuesta Focus pro 3 horas?"

# CONCEPTO -- lo que supon que claude-sonnet-5 decidio bajo cada prompt.
# Bajo v1, la pregunta es pura consulta de precio: get_quote, sin reservar.
decision_v1 = {"tool": "get_quote", "input": {"room": "Focus", "tier": "pro", "hours": 3}}

# Bajo v2, la instruccion de "proactividad" hace que el agente infiera que,
# como ya tiene sala/tier/horas, puede completar la reserva directamente --
# aunque el usuario nunca pidio reservar, solo cotizar.
decision_v2 = {"tool": "book_room", "input": {"room": "Focus", "tier": "pro", "hours": 3, "member": "Guest"}}

print("pregunta:", question)
print("v1 decide:", decision_v1)
print("v2 decide:", decision_v2)
print("misma tool en las dos versiones:", decision_v1["tool"] == decision_v2["tool"])

What to expect:

pregunta: Cuanto cuesta Focus pro 3 horas?
v1 decide: {'tool': 'get_quote', 'input': {'room': 'Focus', 'tier': 'pro', 'hours': 3}}
v2 decide: {'tool': 'book_room', 'input': {'room': 'Focus', 'tier': 'pro', 'hours': 3, 'member': 'Guest'}}
misma tool en las dos versiones: False

Notice what didn't change between the two decisions: the room (Focus), the tier (pro), the hours (3). The agent understood the question perfectly well in both cases — the problem isn't comprehension, it's action. Under v1, understanding "Focus pro 3 hours" means quoting. Under v2, that same understanding, combined with the prompt's new line, means booking — and book_room has real effects: it creates a booking under a made-up "member": "Guest", one the user never authorized.

Why "I tried it by hand" isn't enough

If the person who wrote SYSTEM_PROMPT_V2 tests their change with a single question — say, "Book Focus pro 3h for Ana" — the result looks perfect: the agent books, exactly as expected, because in that case booking was correct. The change looks like an improvement. The problem only shows up with questions that, until now, should never have ended in a booking — and someone testing "by hand" has to guess, from memory, which questions those are and remember to test all of them, every time a line of the prompt changes.

# La prueba manual mas comun: "¿reserva correctamente cuando se lo piden?"
question_reservar = "Reserva Focus pro 3h para Ana"
decision_v2_reservar = {"tool": "book_room", "input": {"room": "Focus", "tier": "pro", "hours": 3, "member": "Ana"}}
print("pregunta:", question_reservar)
print("v2 decide:", decision_v2_reservar)
print("esto SE VE bien -- pero no prueba nada sobre las preguntas de solo precio")

What to expect:

pregunta: Reserva Focus pro 3h para Ana
v2 decide: {'tool': 'book_room', 'input': {'room': 'Focus', 'tier': 'pro', 'hours': 3, 'member': 'Ana'}}
esto SE VE bien -- pero no prueba nada sobre las preguntas de solo precio

This is the exact trap: a manual test confirming the change works for the case you happened to think of says nothing about the cases you didn't test. A fixed set of cases — the one this module picks back up from Module 5 in Lesson 04 — solves that problem by design: it runs every relevant question, always the same ones, every time something changes, without depending on what the person who made the change happens to think of testing.

It isn't just the prompt: tools and schemas get versioned too

This module focuses on the system prompt because it's the change easiest to make "without realizing its magnitude" — it's free text, nobody validates it with a compiler. But the same logic applies to two other changes a real team makes frequently:

• Adding or removing a tool. If book_room stopped being available to the agent (for example, while a production bug gets fixed), any question that used to end in a booking now has no way to complete — a behavior change just as real as the prompt's, even though not a single word of the text got touched.
• Changing an input_schema. If get_quote's tier enum went from ["basic", "pro"] to ["basic", "pro", "premium"], any scripted case assuming only two tiers stops reflecting the contract's reality — the same kind of silent divergence, this time in the contract's shape instead of the prompt's text.

That's why Lesson 03's registry doesn't just store the prompt's text: it also stores a tools version (tools_version) alongside each prompt version, so it stays on record, unambiguously, with exactly which tool set each version ran under.

Common mistakes

1. Thinking a prompt change is "just text," and therefore less risky than a code change. This lesson's experiment shows the opposite: a single-line change in free text silently changed which tool runs for a real question. The tools' code (reservo_tools.py) didn't change a single character.

2. Confusing "the agent responded with no error" with "the agent behaved the same as before." book_room in this lesson's example threw no exception, returned no is_error. It ran perfectly — running the wrong action. A clean log isn't evidence that behavior didn't change.

3. Testing a prompt change only with the questions that "should" change behavior. This lesson's manual test ("book Focus pro 3h for Ana") confirms v2 still books correctly when asked — but it proves nothing about quote-only questions, which are, precisely, the ones that broke.

4. Versioning only the prompt and forgetting tools/schemas. As explained above, adding, removing, or modifying a tool changes the agent's behavior just as much as a text change — and needs to be recorded with the same discipline.

5. Assuming this problem only shows up with large prompt changes. This lesson's change was a single sentence added to a prompt that was otherwise identical. The most dangerous changes aren't the complete redesigns — those get tested carefully, because they're known to be risky — but the small tweaks that feel "safe."

Exercises

Exercise 1: Find the line that changed (Easy)

Without running anything, compare this lesson's SYSTEM_PROMPT_V1 and SYSTEM_PROMPT_V2 word by word and write, in one sentence, exactly which instruction got added. Then, confirm with code that the rest of the text is identical.

v1_words = SYSTEM_PROMPT_V1.split()
v2_words = SYSTEM_PROMPT_V2.split()
solo_en_v2 = [w for w in v2_words if w not in v1_words]
print("palabras que aparecen en v2 y no en v1:", len(solo_en_v2))
print(" ".join(solo_en_v2))

palabras que aparecen en v2 y no en v1: 20
Se proactivo: si ya tienes toda la informacion completar una reserva, complétala directamente en vez cotizar, ahorrarle paso al usuario.

Exercise 2: Design a third question that would also break under v2 (Medium)

The worked example showed "How much does Focus pro 3 hours cost?" breaks under v2. Design another quote-only question (not asking to book) that, through the same mechanism, would also end up booking under v2. Justify why.

question_2 = "Cuanto sale Boardroom pro 2 horas?"
decision_v1_q2 = {"tool": "get_quote", "input": {"room": "Boardroom", "tier": "pro", "hours": 2}}
decision_v2_q2 = {"tool": "book_room", "input": {"room": "Boardroom", "tier": "pro", "hours": 2, "member": "Guest"}}
print("v1:", decision_v1_q2)
print("v2:", decision_v2_q2)

v1: {'tool': 'get_quote', 'input': {'room': 'Boardroom', 'tier': 'pro', 'hours': 2}}
v2: {'tool': 'book_room', 'input': {'room': 'Boardroom', 'tier': 'pro', 'hours': 2, 'member': 'Guest'}}

Exercise 3: Argue why a text diff isn't enough to approve a prompt change (Hard)

Without running anything: write a paragraph explaining why reviewing a system-prompt change by reading its text diff (the way you'd review a code change in a pull-request review) isn't, on its own, enough to decide whether the change is safe. Use this lesson's experiment as evidence.

Summary and next step

• A single-line change to the Reservo agent's system prompt — adding the instruction "be proactive" — changes which tool the agent decides to call for a quote-only question, with no technical error flagging it at all.
• A manual test with the "obvious" question ("book Focus pro 3h for Ana") confirms the change works for that case, but says nothing about the cases that weren't tested — exactly the ones that broke.
• The same risk applies to tools and schemas, not just the prompt's text — that's why Lesson 03's registry versions both.
• The only reliable way to confirm a change is safe is to run a fixed set of representative questions under both versions and compare — the gate this module picks back up from Module 5, starting in Lesson 04. This experiment's question — "How much does Focus pro 3h cost?" — wasn't picked at random for this lesson: it's, literally, quote_focus_pro_3h, one of Module 5's CASE_SET's five fixed cases — Lesson 04 picks this exact case back up, with the same regressed script, and runs it through the real gate.

Next lesson: 03 — The Prompt Version Registry. We build PROMPT_REGISTRY: every version of Reservo's system prompt, identified with a deterministic hash, so there's never any ambiguity about exactly which text was running.

Additional resources

1. Anthropic — System prompts — The system prompt's role in the model's behavior, the piece that changed between v1 and v2 in this lesson.
2. Anthropic — Building effective agents — On why apparently small instructions can have large effects on an agent's decisions.
3. Python — sequence comparison — The foundation of split() and the list comparison used in Exercise 1.
4. Python 3.14 — What's New — The version every line of code in this lesson ran on.