Module 7: Versioning and Safe Rollout

Description

Module 6 closed the harden discipline: a per-tool CircuitBreaker that opens when book_room fails several times in a row, modeled backoff for retries, and the Claude API's own 429 handled with judgment. With that, the Reservo agent survives one tool misbehaving. But there's a completely different failure category no previous module touches: what happens when the entire agent misbehaves, because someone — you, a teammate — changed the system prompt, added a new tool, or tweaked a schema, and that change, without anyone noticing, breaks something that used to work?

That "something breaks without anyone noticing" is, precisely, this module's problem. A circuit breaker doesn't catch it — the tool isn't failing, it's running perfectly, it's just the wrong tool. A structured log doesn't catch it — every event looks clean, with no is_error at all. Cost and latency don't catch it either — an extra booking costs exactly the same, in tokens, as a correct one. What's needed is something that doesn't compare "did it work?" but "did it keep doing what it used to do?" That something is a regression gate, and this module teaches you to use it to decide, with evidence and not with a vibe, whether a new agent version is ready to replace the one already in production.

What this module DOES build

Three pieces, in this order:

A version registry (PROMPT_REGISTRY): the Reservo agent's system prompt, frozen at v1 and v2, each with a deterministic hash that identifies it unambiguously — just like a git commit identifies an exact state of the code, without two people having to "trust" they're looking at the same thing.
A comparison: run the same regression gate — the form one, deterministic, no judge — over v1 and over v2, and see, case by case, whether anything changed.
A decision: GO if v2 passes the gate as well as or better than v1; NO-GO if v2 breaks even a single case v1 used to pass — and, if it's NO-GO, a deterministic rollback back to the version that did work.

You're going to run all three, end to end, over a real case: a v2 of Reservo's system prompt that, with the best intentions in the world ("be more proactive, save steps"), ends up booking a room when the user only asked for the price. The gate catches it. The decision is NO-GO. The rollback goes back to v1. That's the complete module, run with real numbers.

Connection to the module

This module picks back up, without rebuilding anything from scratch, Module 5's form regression gate: a harness that runs a fixed set of cases and compares, literally, against an expected result — never a judge, never a semantic-quality score. Here that same gate gets reused for a new purpose: in Module 5 you ran it once, against the agent exactly as it stood, to confirm it still worked. In this module you run it twice — against the old version and against the new one — and compare the two results against each other. The question changes from "does it work?" to "does it still work the same way?" — and that's, exactly, the question a safe rollout needs to answer before a new version ever talks to a single real user.

Where we are in the ecosystem

Agents in production — operating the Reservo agent
├── Module 1: Why Operating Is Different From Building
├── Module 2: Structured Logging and Tracing a Run
├── Module 3: Measuring Cost and Tokens per Run
├── Module 4: Measuring Latency Honestly
├── Module 5: Regression Evals as a Production Gate
├── Module 6: Failures at Scale — Backoff, Circuit Breakers, and Rate Limits
├── Module 7: Versioning and Safe Rollout  ← YOU ARE HERE
│   → why version, PROMPT_REGISTRY with v1/v2,
│     comparing against the same gate, the GO/NO-GO decision,
│     the deterministic rollback
└── Module 8: Project — the Reservo Agent in Production

This is the fourth and final module in the harden + version discipline, the last of the four organizing this guide: observe (M2) → measure (M3-M4) → gate (M5-M6) → harden and version (M6-M7). With this module closed, Module 8 has all five pieces complete — logging, cost, latency, the gate, the circuit breaker, and now versioning — to operate the Reservo agent end to end, as a single system.

This module's central analogy: a pilot's medical clearance

Before a pilot can fly a passenger plane, they go through a medical exam. It isn't an exam you take once in your life — it repeats, periodically, throughout the pilot's whole career, because the body changes and what was true a year ago may not be true today. The exam is always the same — the same battery of tests, the same blood-pressure, vision, and reflex thresholds — regardless of whether the pilot has been flying for twenty years or this is their first review. If the pilot passes, they fly. If they don't pass, they stay grounded, and another pilot flies instead — one whose clearance is current — until the first one passes the exam again.

A new version of an agent's system prompt is, precisely, that pilot. It doesn't go to production because "it looks good" in a test conversation, or because the person who wrote it trusts it makes things better — it goes to production because it passed the same inspection, with the same thresholds, that the previous version already passed. That exam is Module 5's gate. If the new version fails — if it breaks even a single case the old one used to handle well — it stays grounded: NO-GO, and the previous version, the one with a current clearance, keeps flying. That "the previous version keeps flying" is, exactly, Lesson 07's rollback.

Versioning with no gate is sending the pilot up to fly with no checkup, trusting that "it looks fine." This module exists so that trust is never the criterion.

The case that keeps accompanying the guide: two versions of Reservo, one single question

The agent being versioned in this module is the same as always — the four tools (list_rooms, get_quote, book_room, cancel_booking), the same two price anchors (Focus basic 3h = 7500, Focus pro 3h = 6000). What changes between v1 and v2 is a single thing: the system prompt, the instruction that tells the agent how to behave.

v1 — the original system prompt from agent-fundamentals M8's capstone: uses the tools to quote and book; never books without the user explicitly asking for it.
v2 — the same prompt, with one line added: "be proactive: if you already have all the information needed to complete a booking, complete it directly, to save the user a step." The intent is genuine — fewer turns, less friction. The effect, measured with the gate, is a regression: faced with a question that only asks for a price ("How much does Focus pro 3 hours cost?"), v2 decides to book instead of quote.

This is, deliberately, a subtle regression. There's no syntax error in v2, no KeyError, no tool that stops existing. The agent keeps responding, keeps running real tools, keeps producing results with the correct shape. The only change is which tool it decided to call for a specific question — and that's, precisely, the kind of failure a well-designed form gate is built to catch.

Lesson 02 shows, with a minimal example, why this kind of change is dangerous — and why "I tried it by hand and it looked fine" isn't enough.
Lesson 03 builds PROMPT_REGISTRY: v1 and v2, each with its deterministic prompt_hash (hashlib, never uuid4).
Lesson 04 picks Module 5's gate back up — CASE_SET, check_tool_choice, run_regression_gate, the overrides mechanism — and runs it against both versions: v1 gets PASS (5/5); v2 gets FAIL (4/5), on the quote_focus_pro_3h case.
Lesson 05 opens the gate and shows exactly why it caught this: v2's result shape is perfect — the schema validates — what does go over budget is the chosen tool and, as a side effect, latency (book_room is slower than the get_quote that case expected); the gate needs several independent checks, not just "did something break?"
Lesson 06 builds rollout_decision: the hard rule (never break a case that already passed) and its run over v1 vs. v2 → NO-GO.
Lesson 07 builds rollback: how you go back, deterministically, to the previous version — and why that's simple here, because there's no real deployment mechanism to undo.
Lesson 08 closes the module with a complete ops/versioned_rollout.py: the registry, the comparison, the decision, and the two final artifacts — ops/AGENT_CONFIG.md and AGENT_CHANGELOG.md — genuinely generated.

As in every module of this guide: identifiers and code in English; prose and comments, in Spanish; money in int cents; no random, datetime.now(), or uuid4() — a prompt's hash gets calculated with hashlib, always the same for the same text.

Prerequisites

Required knowledge:

✅ Having completed Module 5 of this guide: the form regression gate — CASE_SET, check_tool_choice, run_regression_gate — deterministic PASS/FAIL, never a judge. This module picks it back up and reproduces it in full (so this module is self-contained), but doesn't re-explain why the gate is about form and not semantics — Module 5 already established that.
✅ Having completed (or knowing well) agent-fundamentals-and-tool-calling-guide M8: the Reservo agent, its four tools, run_reservo_agent, and the notion that the model (claude-sonnet-5) is the piece that decides — concept in this guide, never actually run.
✅ Python: dataclasses, hashlib, dictionaries, dictionary comprehensions, basic pathlib (to write a text file).

NOT required:

❌ You don't need an API key or an internet connection: both system-prompt versions are fixed text; their decisions (which tool they choose) are concept, explicitly scripted.
❌ You don't need a real CI/CD pipeline (GitHub Actions, a deployment system). This module teaches the discipline of comparing two versions against the same gate before deciding — not the infrastructure mechanism that ships code to a server.
❌ You don't need to know anything about A/B testing with real traffic or semantic quality metrics — that's the explicit boundary with evaluation-frameworks-guide, named precisely in Lesson 05.

Environment:

✅ Python 3.14.0 with its standard library (hashlib, dataclasses, pathlib, itertools). Nothing to install.
✅ Modules 2-6's artifacts (observability/, regression/, resilience/), even though this module doesn't import them directly — it builds its own in ops/.

Module roadmap

Lesson 01 — Module introduction (this one)

The limit Module 6 leaves behind — the agent survives a failing tool, but not a silent change in its own behavior — the pilot's medical-clearance analogy, and the map of the eight lessons.

Lesson 02 — Why Version Prompts and Tools

A one-line change in the system prompt can break tool choice without anyone noticing in a manual test. The cost of not versioning: not being able to answer "which exact prompt was running when this happened?"

Lesson 03 — The Prompt Version Registry

PROMPT_REGISTRY: AgentVersion, hash_prompt with hashlib (deterministic, never uuid4), v1 and v2 of Reservo's system prompt, each with its real hash.

Lesson 04 — Comparing a New Version Against the Old

Module 5's gate gets picked back up — CASE_SET, check_tool_choice, run_regression_gate, the overrides mechanism — and run against v1 (PASS 5/5) and against v2 (FAIL 4/5, case quote_focus_pro_3h).

Lesson 05 — The Gate as a Rollout Check

Why the gate caught exactly this: the complete CaseResult for the failed case gets opened, and tool_choice_ok is confirmed as the root cause, with latency_ok broken as a side effect and cost_ok/schema never even evaluated. The boundary with evaluation-frameworks-guide.

Lesson 06 — Go or No-Go

rollout_decision: the rule (never break a case the old version passed), run over v1 vs. v2 → NO-GO, and over v1 vs. a fixed v3 → GO.

Lesson 07 — Rolling Back

rollback: going back, deterministically, to the previous version when the decision is NO-GO. Why it's simple here — it's a pointer change, not a real deployment.

Lesson 08 — Mini-Project: A Versioned Rollout for Reservo

A complete ops/versioned_rollout.py: registry, gate over both versions, decision, rollback, and the two final artifacts genuinely generated — AGENT_CONFIG.md and AGENT_CHANGELOG.md.

Progression map

Lesson 01 (this)  → Module 6's limit, the pilot's medical clearance
Lesson 02         → Why version: the danger of a silent change
Lesson 03         → PROMPT_REGISTRY, AgentVersion, hash_prompt
Lesson 04         → Module 5's gate, run on v1 and on v2
Lesson 05         → Why the gate caught exactly this: tool vs. schema vs. latency
Lesson 06         → rollout_decision: GO / NO-GO
Lesson 07         → rollback: going back to the version that worked
Lesson 08         → Mini-project: the complete versioned rollout

Difficulty: ⭐⭐⭐ ──────────────────▶ ⭐⭐⭐

What you'll achieve in this module

By completing the 8 lessons, you'll be able to:

Explain, with a concrete example, why a system-prompt change can break tool choice without any explicit error flagging it — no KeyError, no is_error, no invalid schema.
Build a version registry (PROMPT_REGISTRY) that identifies each prompt version with a deterministic hash, never with an identifier that changes between runs.
Run the same regression gate against two different agent versions and read, case by case, where behavior diverges.
Precisely tell apart which part of a form check caught a regression — chosen tool, output schema, or latency budget — instead of treating the gate as a black box that says "something broke."
Make a GO/NO-GO decision with an explicit, verifiable rule, never with the subjective impression that "the new version looks better."
Run a deterministic rollback that goes back to the previous version, and explain why this guide treats it as a pointer change, not an infrastructure deployment.

Before and after

BEFORE the module:
→ "I tried the new prompt version with two questions and it
  looked fine, so I'm shipping it"
→ "if the agent doesn't throw any error, it's working the same
  as before"
→ "versioning a prompt is saving the text in a file, that's it"
→ "reverting a bad change is pasting the old prompt back in
  by hand"

AFTER the module:
→ a new version only goes to production if it passes the SAME
  gate, with the SAME thresholds, the old version already passed
→ an agent can keep responding with no technical error at all
  and still have changed WHAT it does -- the gate compares
  behavior, not just the absence of exceptions
→ versioning is a registry with a hash that unambiguously
  identifies the exact text that was running
→ a rollback is a ONE-line decision (go back to the previous
  hash), made with evidence, not an "undo by hand"

Traps to avoid in this module

1. "If `v2` responds well to the questions I tried by hand, it's ready"

No. This module's central example is, exactly, a version that responds "well" — with no technical error at all — and is still broken. Testing "by hand" with two or three questions never replaces running the same fixed case set already used to validate the previous version.

2. "This module's gate needs an LLM to compare both versions and say which is better"

No. The gate is still exactly Module 5's: a literal comparison against a fixed value (which tool got called, what it returned). There's never a call to a model to grade anything — that's, precisely, the boundary with evaluation-frameworks-guide, named in Lesson 05.

3. "A prompt hash needs a special library or an external service"

No. The standard library's hashlib.sha256, applied to the prompt's text, is deterministic and sufficient: the same text always produces the same hash, on any machine, with no network and no external dependency at all.

4. "GO/NO-GO is a business decision, not something you can calculate"

You can, and this module does: the rule is explicit — v2 can never break a case v1 used to pass — and it's calculated by comparing two GateReports case by case. It's reasonable for a person to confirm the final decision; it isn't reasonable for the evidence behind it to be subjective.

5. "This module's rollback is the same as a real infrastructure rollback (Kubernetes, a load balancer)"

No, and Lesson 07 says so precisely: here rollback is a pointer change — which prompt version is active — because there's no real deployment to undo. A complete infrastructure rollback (containers, traffic, DNS) is a different layer's topic, outside this guide's $0 scope.

How to work through this module

Run the gate against both versions yourself, before reading the result in Lesson 04. The surprise of seeing FAIL 4/5 instead of PASS 5/5 is more useful than reading it already solved.
In Lesson 05, don't settle for "the gate failed" — open the complete CaseResult and confirm, with your own eyes, which of its five fields is the one that genuinely broke.
The mini-project (Lesson 08) is the synthesis. There you're going to genuinely generate AGENT_CONFIG.md and AGENT_CHANGELOG.md — the same two artifacts Module 8 is going to cite in the guide's final capstone.

Estimated time:

Lesson 01 (this)  →  20 min reading
Lesson 02         →  20 min + running the example
Lesson 03         →  20 min + running the example
Lesson 04         →  30 min + running the example
Lesson 05         →  25 min + running the example
Lesson 06         →  25 min + running the example
Lesson 07         →  20 min + running the example
Lesson 08         →  35 min + building the complete mini-project

Total: ~3.5 hours

Evidence of success

Before moving on to Module 8 (Project: the Reservo Agent in Production), you should be able to:

✅ Explain, with this module's v2 case, why an agent can keep "working" with no technical error at all and still have changed its behavior in a way that matters.
✅ Build a PROMPT_REGISTRY with deterministic hashes, and explain why uuid4() is never used to identify a version.
✅ Run the regression gate against two versions and read, precisely, on which case they diverge.
✅ Make a GO/NO-GO decision applying the explicit rule, not an impression.
✅ Run a deterministic rollback and explain why, in this guide, it's a pointer change and not a deployment.

Summary

This module closes the harden + version discipline: after Module 6 hardened the agent against a failing tool, this module hardens it against itself — against a change to its own prompt that breaks something without warning.
The central analogy is the pilot's medical clearance: a new version doesn't fly until it passes the same inspection — Module 5's gate — the old one already passed; if it fails, it stays grounded (NO-GO) and the previous version keeps flying (rollback).
The module's pieces: why version (L02), the hash-based registry (L03), comparing against the same gate (L04), why the gate caught exactly this (L05), the GO/NO-GO decision (L06), the rollback (L07), and the mini-project that brings it all together (L08).
Run end to end: v1 passes the gate PASS (5/5); v2 — with a single new line in the prompt — fails FAIL (4/5), exactly on the quote_focus_pro_3h case (Focus pro 3h); the decision is NO-GO; the rollback returns the agent to v1.

Next lesson: 02 — Why Version Prompts and Tools. Before building any registry, we see with a minimal example why a single-line change in the system prompt can break tool choice without any manual test noticing.

Additional resources

Anthropic — Building effective agents — On why apparently small changes to an agent's instructions can have large, non-obvious effects on its behavior.
Anthropic — System prompts — The system prompt's role in the model's behavior, the piece this module versions.
Python — hashlib — hashlib.sha256, the deterministic core of hash_prompt in Lesson 03.
Python 3.14 — What's New — The version all of this module's engineering runs on.

← PreviousMini-Project: A Resilient Reservo Agent Next →Why Version Prompts and Tools