Mini-Project: A Versioned Rollout for Reservo

Description

Seven lessons built, separately, each piece: why version (02), the deterministic-hash registry (03), Module 5's gate run against two versions via overrides (04), the breakdown of which check catches which type of regression (05), the GO/NO-GO decision with its strict rule (06), and the deterministic rollback (07). This mini-project brings them all together into ops/versioned_rollout.py, a single script running the complete cycle end to end: it registers the versions, runs regression/harness.py's gate (Module 5, untouched) on both, decides, acts on the decision, and genuinely generates this module's two final artifacts — AGENT_CONFIG.md and AGENT_CHANGELOG.md.

By the end of this lesson you're going to have this guide's sixth real artifact — after observability/run_logger.py, observability/cost_calculator.py, observability/latency_model.py, regression/harness.py+regression/golden_cases.json, and resilience/tool_circuit_breaker.py — ready for Module 8 to cite, with real output, in the final capstone that operates the complete Reservo agent.

Connection to the module

This is the synthesis of the eight lessons. There's no new mechanism piece — PROMPT_REGISTRY, run_regression_gate (reused unchanged from Module 5), rollout_decision, rollback are exactly Lessons 03 through 07's; this mini-project's job is assembling them into a single flow and running them together, over the same real scenario accompanying the entire module: v2 with its quote_focus_pro_3h regression, caught by the gate, resolved with a NO-GO and a rollback to v1.

ops/versioned_rollout.py, complete

# ops/versioned_rollout.py
"""Rollout versionado del agente de Reservo (Módulo 7): registro de
versiones con hash determinista (L03), el gate de regresión del Módulo 5
—reusado sin cambios— corrido en dos versiones vía `overrides` (L04-L05),
la decisión GO/NO-GO (L06), y el rollback (L07). Genera AGENT_CONFIG.md y
AGENT_CHANGELOG.md."""
from pathlib import Path

from ops.versions.prompt_registry import PROMPT_REGISTRY
from regression.harness import load_case_set, run_regression_gate


def compare_versions(case_set, overrides_old, overrides_new):
    """L04: envoltura delgada -- corre run_regression_gate dos veces, una
    por version, y devuelve ambos GateReport sin decidir nada todavia."""
    gate_old = run_regression_gate(case_set, overrides=overrides_old)
    gate_new = run_regression_gate(case_set, overrides=overrides_new)
    return gate_old, gate_new


# El guion que v2 produciría para "¿Cuánto cuesta Focus pro 3h?" -- el mismo
# guion regresivo que el Módulo 5 (lección 05) ya usó como su ejemplo
# canónico de FAIL, reusado aquí sin cambios (L04).
V2_REGRESSED_SCRIPT = [
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "book_room",
         "input": {"room": "Focus", "tier": "pro", "hours": 3, "member": "Ana"}}]},
    {"stop_reason": "end_turn", "content": [
        {"type": "text", "text": "Reservé Focus pro 3h para Ana."}]},
]

VERSION_OVERRIDES = {
    "v1": {},
    "v2": {"quote_focus_pro_3h": V2_REGRESSED_SCRIPT},
}


def rollout_decision(gate_old, gate_new):
    """L06: GO si la version nueva pasa el gate igual o mejor que la vieja
    -- NUNCA puede romper un caso que la vieja pasaba."""
    old_by_name = {c.name: c for c in gate_old.cases}
    new_by_name = {c.name: c for c in gate_new.cases}
    broken = [
        name for name, old_c in old_by_name.items()
        if old_c.passed and not new_by_name[name].passed
    ]
    return ("NO-GO", broken) if broken else ("GO", [])


def rollback(active_version_id, previous_version_id, registry):
    """L07: un cambio de puntero determinista -- v1 nunca se borro del
    registro, asi que 'volver' no reconstruye nada."""
    if previous_version_id not in registry:
        raise KeyError(f"version desconocida en el registro: {previous_version_id!r}")
    return previous_version_id


def write_agent_config(path, active_version_id, registry):
    """El estado actual del rollout: que version esta activa, y el
    registro completo de las que existen."""
    lines = [f"# AGENT_CONFIG -- version activa: {active_version_id}", ""]
    for version_id, av in registry.items():
        marker = " (ACTIVA)" if version_id == active_version_id else ""
        lines.append(f"## {version_id}{marker}")
        lines.append(f"- hash: `{av.prompt_hash}`")
        lines.append(f"- tools_version: `{av.tools_version}`")
        lines.append(f"- model: `{av.model}`")
        lines.append(f"- nota: {av.note}")
        lines.append("")
    path.write_text("\n".join(lines))


def append_changelog(path, entry):
    """El historial de decisiones de rollout: se agrega, nunca se
    sobrescribe -- cada intento de version queda documentado."""
    existing = path.read_text() if path.exists() else "# AGENT_CHANGELOG\n\n"
    path.write_text(existing + entry + "\n")


def main():
    case_set = load_case_set("regression/golden_cases.json")
    v1 = PROMPT_REGISTRY["v1"]
    v2 = PROMPT_REGISTRY["v2"]

    print("--- registro de versiones ---")
    for version_id, av in PROMPT_REGISTRY.items():
        print(f"{version_id}  hash={av.prompt_hash}  tools={av.tools_version}  model={av.model}")
    print()

    report_v1, report_v2 = compare_versions(case_set, VERSION_OVERRIDES["v1"], VERSION_OVERRIDES["v2"])
    print(f"v1: {'PASS' if report_v1.passed else 'FAIL'} "
          f"({sum(c.passed for c in report_v1.cases)}/{len(report_v1.cases)})")
    fails = [c.name for c in report_v2.cases if not c.passed]
    print(f"v2: {'PASS' if report_v2.passed else 'FAIL'} "
          f"({sum(c.passed for c in report_v2.cases)}/{len(report_v2.cases)}) -- fallan: {fails}")
    print()

    active_version = "v1"
    decision, broken = rollout_decision(report_v1, report_v2)
    print(f"rollout_decision(v1, v2) -> {decision}, casos_rotos={broken}")

    if decision == "GO":
        active_version = "v2"
        changelog_entry = (
            f"## v2 ({v2.prompt_hash})\n"
            f"- Gate: PASS ({sum(c.passed for c in report_v2.cases)}/{len(report_v2.cases)})\n"
            f"- Decision: GO -- version activa actualizada a v2\n"
        )
    else:
        active_version = rollback(active_version, "v1", PROMPT_REGISTRY)
        broken_case = next(c for c in report_v2.cases if not c.passed)
        expected_case = next(c for c in case_set if c["name"] == broken_case.name)
        changelog_entry = (
            f"## v2 ({v2.prompt_hash})\n"
            f"- Gate: FAIL ({sum(c.passed for c in report_v2.cases)}/{len(report_v2.cases)}) "
            f"en {broken_case.name} (tool esperada={expected_case['expected_tools']}, "
            f"obtenida={broken_case.actual_tools})\n"
            f"- Decision: NO-GO -- rollback a v1 ({v1.prompt_hash})\n"
        )

    print("version activa final:", active_version)
    print()

    out_dir = Path(".")
    write_agent_config(out_dir / "AGENT_CONFIG.md", active_version, PROMPT_REGISTRY)
    append_changelog(out_dir / "AGENT_CHANGELOG.md", changelog_entry)

    print("--- AGENT_CONFIG.md ---")
    print((out_dir / "AGENT_CONFIG.md").read_text())
    print("--- AGENT_CHANGELOG.md ---")
    print((out_dir / "AGENT_CHANGELOG.md").read_text())


if __name__ == "__main__":
    main()

Running the complete rollout

main()

What to expect:

--- registro de versiones ---
v1  hash=c5757b6d6264  tools=tools-v1  model=claude-sonnet-5
v2  hash=c364e85e5649  tools=tools-v1  model=claude-sonnet-5

v1: PASS (5/5)
v2: FAIL (4/5) -- fallan: ['quote_focus_pro_3h']

rollout_decision(v1, v2) -> NO-GO, casos_rotos=['quote_focus_pro_3h']
version activa final: v1

--- AGENT_CONFIG.md ---
# AGENT_CONFIG -- version activa: v1

## v1 (ACTIVA)
- hash: `c5757b6d6264`
- tools_version: `tools-v1`
- model: `claude-sonnet-5`
- nota: System prompt original del capstone de agent-fundamentals M8.

## v2
- hash: `c364e85e5649`
- tools_version: `tools-v1`
- model: `claude-sonnet-5`
- nota: Agrega una instruccion de proactividad para reducir turnos.

--- AGENT_CHANGELOG.md ---
# AGENT_CHANGELOG

## v2 (c364e85e5649)
- Gate: FAIL (4/5) en quote_focus_pro_3h (tool esperada=['get_quote'], obtenida=['book_room'])
- Decision: NO-GO -- rollback a v1 (c5757b6d6264)

Every line of this output traces back to a specific lesson in the module. The registry (Lesson 03) identifies v1 and v2 with no ambiguity, with their real hashes. The gate (Lesson 04) — reused, untouched, from Module 5 — confirms PASS (5/5) against FAIL (4/5), with quote_focus_pro_3h flagged precisely — the same case whose complete breakdown Lesson 05 opened in depth. rollout_decision (Lesson 06) translates that evidence into NO-GO. rollback (Lesson 07) returns the active version to v1. And the two generated files — AGENT_CONFIG.md, with the system's current state; AGENT_CHANGELOG.md, with the decision's history and its reason — are real artifacts on disk, not just text printed to a terminal: anyone on the team, days later, can open AGENT_CONFIG.md and know, without asking anyone, that the active version is v1 and why v2 never reached production.

The opposite scenario: if v2 had passed

It's worth confirming, by running it, that the same script produces the opposite result if the evidence were different — for example, if the team fixed the proactivity instruction and the complete CASE_SET passed again with no substitution at all (Lesson 06's v3):

report_v3 = run_regression_gate(case_set, overrides={})  # v3: corrige la regresión de v2
decision_v3, broken_v3 = rollout_decision(report_v1, report_v3)
print(f"v3: {'PASS' if report_v3.passed else 'FAIL'} ({sum(c.passed for c in report_v3.cases)}/{len(report_v3.cases)})")
print(f"rollout_decision(v1, v3) -> {decision_v3}, casos_rotos={broken_v3}")

What to expect:

v3: PASS (5/5)
rollout_decision(v1, v3) -> GO, casos_rotos=[]

main() with this decision would have taken the if's other branch: active_version = "v2" (or, in this case, "v3"), and AGENT_CHANGELOG.md would have documented a GO instead of a NO-GO — the same script, with not a single different line, following the evidence wherever it leads.

Common mistakes

1. Running main() twice in a row without deleting AGENT_CHANGELOG.md and expecting the same file. append_changelog appends, it doesn't overwrite — running main() twice produces two ## v2 (...) entries in the same file. This is intentional (the changelog is a cumulative history), but it can surprise someone expecting an identical file on every run.

2. Confusing AGENT_CONFIG.md (the current state) with AGENT_CHANGELOG.md (the history). write_agent_config overwrites completely on every run — it always reflects the active version right now — append_changelog accumulates — every run adds a new entry, without deleting previous ones. They're two artifacts with different purposes, and confusing which one overwrites and which one accumulates is an easy mistake to make reading the code for the first time.

3. Thinking this mini-project rebuilds Module 5's gate. It doesn't rebuild it — it imports and reuses it, twice, against two different overrides. run_regression_gate is, literally, the same function from regression/harness.py; this script never redefines run_case, check_tool_choice, or any other check.

4. Forgetting next(c for c in report_v2.cases if not c.passed) assumes there's exactly one broken case. In this specific scenario (v2 with a single regression) it works with no problem, but if a future change broke two or more cases (like Lesson 04's Exercise 2), this line would only capture the first one it finds — a complete AGENT_CHANGELOG.md, in that scenario, would need to iterate over all the broken cases, not just take the first.

5. Running this script without having run the gate against v1 alone first, as a baseline. As Lesson 04 warned, without v1's PASS (5/5) baseline, v2's FAIL (4/5) has nothing to compare against — main() always runs both before deciding, precisely so it never depends on a baseline assumed from memory.

Exercises

Exercise 1: Confirm AGENT_CONFIG.md reflects state, not history (Easy)

Run main() twice in a row, without changing anything. Confirm AGENT_CONFIG.md has exactly the same content after both runs (because it always gets overwritten with the current state), while AGENT_CHANGELOG.md grows (because it accumulates).

main()
config_despues_de_1 = Path("AGENT_CONFIG.md").read_text()
changelog_despues_de_1 = Path("AGENT_CHANGELOG.md").read_text()

main()
config_despues_de_2 = Path("AGENT_CONFIG.md").read_text()
changelog_despues_de_2 = Path("AGENT_CHANGELOG.md").read_text()

print("AGENT_CONFIG.md identico entre corridas:", config_despues_de_1 == config_despues_de_2)
print("AGENT_CHANGELOG.md crecio:", len(changelog_despues_de_2) > len(changelog_despues_de_1))
print("lineas con '## v2' en el changelog final:", changelog_despues_de_2.count("## v2"))

AGENT_CONFIG.md identico entre corridas: True
AGENT_CHANGELOG.md crecio: True
lineas con '## v2' en el changelog final: 2

Exercise 2: Extend write_agent_config to include a run counter (Medium)

The current AGENT_CONFIG.md doesn't say how many times the gate ran. Without using datetime.now() (banned in any executed code block in this guide), add a gate_run_sequence parameter to write_agent_config — an integer counter representing "the run number," not a real date — and add it as one more line in the generated file.

def write_agent_config_v2(path, active_version_id, registry, gate_run_sequence):
    """Version extendida: agrega un contador de corrida determinista, NUNCA
    una fecha real (prohibido datetime.now() en esta guia)."""
    lines = [
        f"# AGENT_CONFIG -- version activa: {active_version_id}",
        f"(corrida de gate numero {gate_run_sequence})",
        "",
    ]
    for version_id, av in registry.items():
        marker = " (ACTIVA)" if version_id == active_version_id else ""
        lines.append(f"## {version_id}{marker}")
        lines.append(f"- hash: `{av.prompt_hash}`")
        lines.append(f"- tools_version: `{av.tools_version}`")
        lines.append(f"- model: `{av.model}`")
        lines.append(f"- nota: {av.note}")
        lines.append("")
    path.write_text("\n".join(lines))

write_agent_config_v2(Path("AGENT_CONFIG.md"), "v1", PROMPT_REGISTRY, gate_run_sequence=1)
print(Path("AGENT_CONFIG.md").read_text().split("\n\n")[0])

# AGENT_CONFIG -- version activa: v1
(corrida de gate numero 1)

Exercise 3: Simulate a complete cycle of two consecutive rollouts (Hard)

Simulate the following complete sequence: (1) v1 active, v2 gets evaluated, NO-GO, rollback to v1; (2) with v1 still active, a fixed v3 gets evaluated (overrides={}, no substitution at all), GO, gets promoted to v3. Calculate v3's real prompt_hash with hash_prompt over a fixed prompt text (don't make it up by hand), register it in PROMPT_REGISTRY, and finally print the resulting AGENT_CONFIG.md — confirm it reflects v3 as the active version, with v1 and v2 still documented (but not active) in the same file.

from ops.versions.prompt_registry import hash_prompt, AgentVersion

active_version = "v1"

# Paso 1: v2 falla, rollback a v1 (ya ejecutado por main() arriba)
decision_1, broken_1 = rollout_decision(report_v1, report_v2)
if decision_1 == "NO-GO":
    active_version = rollback(active_version, "v1", PROMPT_REGISTRY)
print("despues del intento con v2:", active_version)

# Paso 2: v3 (corregida) pasa, se promueve
report_v3 = run_regression_gate(case_set, overrides={})
decision_2, broken_2 = rollout_decision(report_v1, report_v3)
if decision_2 == "GO":
    active_version = "v3"
print("despues del intento con v3:", active_version)

SYSTEM_PROMPT_V3 = (
    "Eres el asistente de reservas de Reservo, un sistema de coworking. "
    "Ayudas a los usuarios a consultar salas, cotizar precios, reservar y "
    "cancelar reservas. Usa siempre las tools disponibles para cotizar y "
    "reservar -- nunca inventes un precio de memoria. Cuando el usuario "
    "pregunta un precio, usa get_quote y NO reserves, incluso si podrias "
    "inferir todos los datos necesarios para reservar. Usa book_room "
    "unicamente cuando el usuario pide reservar de forma explicita."
)
PROMPT_REGISTRY["v3"] = AgentVersion(
    version_id="v3", prompt_text=SYSTEM_PROMPT_V3,
    prompt_hash=hash_prompt(SYSTEM_PROMPT_V3), tools_version="tools-v1",
    model="claude-sonnet-5", note="Corrige la regresion de v2 en quote_focus_pro_3h.",
)
write_agent_config(Path("AGENT_CONFIG.md"), active_version, PROMPT_REGISTRY)
print()
print(Path("AGENT_CONFIG.md").read_text())

despues del intento con v2: v1
despues del intento con v3: v3

# AGENT_CONFIG -- version activa: v3

## v1
- hash: `c5757b6d6264`
- tools_version: `tools-v1`
- model: `claude-sonnet-5`
- nota: System prompt original del capstone de agent-fundamentals M8.

## v2
- hash: `c364e85e5649`
- tools_version: `tools-v1`
- model: `claude-sonnet-5`
- nota: Agrega una instruccion de proactividad para reducir turnos.

## v3 (ACTIVA)
- hash: `c5c4c4631f7e`
- tools_version: `tools-v1`
- model: `claude-sonnet-5`
- nota: Corrige la regresion de v2 en quote_focus_pro_3h.

Summary and next step

• We assembled a complete ops/versioned_rollout.py: PROMPT_REGISTRY (L03), run_regression_gate reused unchanged from Module 5 and applied to two different overrides (L04-L05), rollout_decision (L06), and rollback (L07) — seven lessons, one single script.
• We ran it over this module's central scenario: v1 passes the gate PASS (5/5); v2 — with a single new line in its system prompt — fails FAIL (4/5), exactly on quote_focus_pro_3h; the decision is NO-GO; the rollback returns the active version to v1.
• We genuinely generated the two final artifacts: AGENT_CONFIG.md (the current state, overwritten on every run) and AGENT_CHANGELOG.md (the accumulated decision history, with the exact reason for each one, citing the gate's literal message: expected tool, tool obtained).
• We confirmed, running the opposite scenario with a fixed v3, that the same script produces GO when the evidence backs it up — this module's discipline never depends on which version is being evaluated, only on what Module 5's gate confirms.

This closes Module 7. You have a complete ops/versioned_rollout.py, and run evidence that a single-line change to a system prompt — with the best intentions in the world — can break an agent's tool choice with no visible technical error at all, and that Module 5's same form gate, run twice with discipline, catches it before it ever talks to a real user.

Next module: Module 8 — Project: the Reservo Agent in Production. This guide's capstone takes agent-fundamentals M8's agent and operates it complete: logging and trace (M2), measured cost and latency (M3-M4), the regression gate run against it (M5), a circuit breaker over book_room with a simulated failure (M6), and this same rollout comparison between two versions of its config (M7) — the four final deliverables, cited with real output: RUN_LOG.jsonl, the metrics summary, regression_report.json, and AGENT_CHANGELOG.md.

Additional resources

1. Anthropic — Building effective agents — On why the discipline of operating an agent — measuring, gating, versioning — is just as important as building it well the first time.
2. Anthropic — System prompts — The piece this entire module versioned, compared, and decided to keep or revert.
3. Python — pathlib — Path.write_text and Path.read_text, used to generate AGENT_CONFIG.md and AGENT_CHANGELOG.md as real files on disk.
4. Python — hashlib — The deterministic core of this module's entire version registry.
5. Python 3.14 — What's New — The version every line of code in this module ran on, including this mini-project's final report.