The Fixed Case Set

Description

Lesson 02 confirmed, in isolation, that check_schema, check_tool_choice, check_cost_threshold, and check_latency_threshold work over handwritten data. A real regression gate, though, doesn't run over a single improvised case each time — it always runs over the same set of cases, one after another, every time something in the system changes. This lesson builds that set: regression/golden_cases.json, five fixed cases covering Reservo's four tools, each with its question, its turn script, and the exact expected result.

This lesson's keyword is fixed. There's no mechanism, in any file in this module, that generates a new case, tweaks one "a little" to vary the test, or deletes it after running it. The same file, run today, run a year from now, produces the same question, the same script, and the same comparison — exactly the property that makes a regression gate reliable.

Connection to the module

This lesson delivers a complete regression/golden_cases.json — this module's second new artifact, alongside regression/harness.py — and reset_reservo_state, the function that guarantees no case in the set depends on the order the others ran in. Lessons 04 through 07 run, over and over, exactly these five cases.

Why a fixed set, and not cases generated on the fly

Generating new cases every time the gate runs — varying the hours, the member's name, the room — might seem more "thorough." This guide, as in every module before it, rejects that idea for the same reason as always: reproducibility. If the CASE_SET changed from one run to the next, today's FAIL could turn into tomorrow's PASS with nothing in the system having changed — simply because the case that got randomly generated this time turned out easier — and today's PASS could hide a real regression that a differently generated case would have caught. A gate whose result depends on which case it happened to run isn't a gate — it's a lottery with a passed sticker slapped on top.

A fixed CASE_SET has the reverse property: if the gate passed yesterday and fails today, the only possible explanation is that something in the system changed — never that the case was different. That's, precisely, the property that makes a FAIL useful information, instead of noise.

The shape of a case

Each entry in golden_cases.json is a JSON object with seven fields, all already familiar data from this guide — the same question/model_script format every previous lesson uses for the model's script (concept) — plus three new fields declaring this module's exact expectation:

{
  "name": "quote_focus_pro_3h",
  "question": "¿Cuánto cuesta Focus pro 3h?",
  "model_script": [
    {"stop_reason": "tool_use", "content": [
      {"type": "tool_use", "id": "toolu_01", "name": "get_quote",
       "input": {"room": "Focus", "tier": "pro", "hours": 3}}]},
    {"stop_reason": "end_turn", "content": [
      {"type": "text", "text": "Focus pro 3h cuesta $60.00."}]}
  ],
  "expected_tools": ["get_quote"],
  "expected_output": {"price_cents": 6000},
  "cost_threshold_cents": 5,
  "latency_threshold_ms": 100
}

• name — a readable identifier, unique within the CASE_SET, that's going to show up in every PASS/FAIL message in the lessons that follow.
• question and model_script — exactly what run_reservo_agent needs to run this case: the question and the model's turn script (concept), in the same format as always.
• expected_tools — the literal tool sequence check_tool_choice (lesson 05) is going to require.
• expected_output — a dict of key-value pairs the last called tool's result has to match exactly — this case's price_cents: 6000 anchor is the same Focus-pro-3h anchor that's been accompanying this guide since Module 1.
• cost_threshold_cents and latency_threshold_ms — the fixed thresholds check_cost_threshold/check_latency_threshold (lesson 06) are going to apply to this specific case.

This module's complete CASE_SET has five cases, each anchored to a different fragment of Reservo's behavior: two quotes (one per tier, anchored to the same two price anchors as always — 7500 for basic, 6000 for pro), two complete bookings (list_rooms → get_quote → book_room, over two different rooms), and one booking followed by a cancellation.

load_case_set: reading the file, with no surprises

import json


def load_case_set(path):
    with open(path, encoding="utf-8") as fh:
        return json.load(fh)

There's nothing worth highlighting in this function — and that's the point. golden_cases.json is pure JSON: every value inside model_script (the tool_use/tool_result/text blocks, with their strings, integers, and booleans) is exactly the same kind of structure you already used, with no modification, to write a turn script by hand in any earlier lesson of this guide. No special parser is needed, no custom class — json.load is enough because the CASE_SET never needed to be more than data.

CASE_SET = load_case_set("golden_cases.json")
print("casos cargados:", len(CASE_SET))
print()
for c in CASE_SET:
    print(f"{c['name']:38} tools={c['expected_tools']}  cost<=+{c['cost_threshold_cents']}c  latency<={c['latency_threshold_ms']}ms")

What to expect:

casos cargados: 5

quote_focus_pro_3h                     tools=['get_quote']  cost<=+5c  latency<=100ms
quote_focus_basic_3h                   tools=['get_quote']  cost<=+5c  latency<=100ms
book_focus_pro_3h_ana                  tools=['list_rooms', 'get_quote', 'book_room']  cost<=+5c  latency<=250ms
book_boardroom_pro_1h_sofia            tools=['list_rooms', 'get_quote', 'book_room']  cost<=+5c  latency<=250ms
book_and_cancel_studio_basic_1h_diego  tools=['book_room', 'cancel_booking']  cost<=+5c  latency<=250ms

Also confirm the file is valid JSON end to end — reading it and re-serializing it produces exactly the same structure:

raw = open("golden_cases.json", encoding="utf-8").read()
reparsed = json.loads(raw)
print("round-trip exacto:", reparsed == CASE_SET)
print("bytes del archivo:", len(raw))

round-trip exacto: True
bytes del archivo: 3923

reset_reservo_state: isolating each case from the one that ran before

reservo_tools.py has shared module-level state — BOOKINGS, the bookings dictionary, and _booking_ids, the counter that assigns each booking_id. That state is exactly what makes book_room work (every booking needs a unique id), but it's also a real trap for a CASE_SET with more than one booking case: if two cases run, one after another, in the same Python process, with no reset in between, the second case inherits the first one's counter — its booking_id would no longer be the 1 the case declares in its expected_output, but whatever the counter has reached.

import itertools
import reservo_tools as rt


def reset_reservo_state():
    """Resetea el estado compartido de Reservo (BOOKINGS + el contador de
    ids) antes de CADA caso, para que ningún caso dependa del orden en que
    corrieron los demás -- la misma disciplina de aislamiento de cualquier
    suite de tests."""
    rt.BOOKINGS.clear()
    rt._booking_ids = itertools.count(1)

Confirm the problem, and the fix, with the CASE_SET's two booking cases — Ana and Sofía:

def booking_id_of(case, sequence_number):
    reset_reservo_state()
    with rl.traced_run(case["question"], sequence_number) as trace_id:
        final, history = ra.run_reservo_agent(case["question"], case["model_script"])
    result = json.loads(_last_result_for_tool(history, "book_room"))  # helper de la lección 04
    return result["booking_id"]

case_ana = CASE_SET[2]      # book_focus_pro_3h_ana
case_sofia = CASE_SET[3]    # book_boardroom_pro_1h_sofia

print("booking_id de Ana   :", booking_id_of(case_ana, 1))
print("booking_id de Sofía :", booking_id_of(case_sofia, 2))

What to expect:

booking_id de Ana   : 1
booking_id de Sofía : 1

Both cases get booking_id: 1 — matching what each one declares in its expected_output, regardless of the order they ran in. Now, the same pair of cases, but without reset_reservo_state between them:

reset_reservo_state()
with rl.traced_run(case_ana["question"], 1) as trace_id:
    final, history = ra.run_reservo_agent(case_ana["question"], case_ana["model_script"])
with rl.traced_run(case_sofia["question"], 2) as trace_id2:
    final2, history2 = ra.run_reservo_agent(case_sofia["question"], case_sofia["model_script"])  # SIN reset
result2 = json.loads(_last_result_for_tool(history2, "book_room"))
print("SIN reset entre casos, booking_id de Sofía:", result2["booking_id"])

SIN reset entre casos, booking_id de Sofía: 2

Without the reset, Sofía's case inherits the counter Ana's case left behind, and its real booking_id (2) no longer matches the 1 its expected_output declares — a FAIL that would have absolutely nothing to do with any real change in Reservo's behavior, only with the order the cases ran in. That kind of FAIL — produced by the harness itself, not by the system under evaluation — is exactly what reset_reservo_state, run at the start of every case, eliminates at the root. This is the same isolation discipline of any real test suite: every case has to be able to run alone, or alongside any other, in any order, and always produce the same result.

Common mistakes

1. Writing expected_tools as a set instead of a list, "because order shouldn't matter." check_tool_choice (lesson 05) compares with == against an ordered list — order does matter, on purpose: an agent that calls book_room before get_quote has a real problem, even if it ends up calling the same two tools. A set would lose that information.

2. Forgetting reset_reservo_state and blaming the agent for a FAIL the harness itself actually produced. This lesson's worked example demonstrates it with numbers: without the reset, the CASE_SET's second booking case fails its expected_output with nothing in Reservo having changed. Before investigating a supposed regression, always confirm the harness is resetting state correctly.

3. Modifying golden_cases.json "by hand, for one quick test" and forgetting to revert it. Like any fixed file, an uncontrolled change stops it from being fixed — and whoever runs the gate next is going to be comparing against a different expectation than any earlier run used, with no record of what changed or why.

4. Thinking reset_reservo_state also resets run_logger._sequence or the trace_id. It doesn't, and it shouldn't: the log sequence counter and each case's trace_id are observability identifiers (Module 2), independent of Reservo's business state (BOOKINGS). Mixing both resets would be a layer confusion — each one lives in its own file, with its own responsibility.

5. Adding a sixth case to the CASE_SET without declaring all seven complete fields. A case without cost_threshold_cents, for example, doesn't produce a visible error when the JSON loads — json.load doesn't validate any of this — but it does produce a confusing KeyError later on, in lesson 07, when the harness tries to read a field that isn't there. This lesson doesn't build that validation yet (check_schema could, in principle, be applied to the CASE_SET itself — an idea Exercise 3 explores).

Exercises

Exercise 1: Count how many cases expect each tool (Easy)

Without opening the file in a text editor, use CASE_SET (already loaded in Python) to count how many of the five cases have get_quote appear somewhere in expected_tools, and how many have cancel_booking.

quote_count = sum(1 for c in CASE_SET if "get_quote" in c["expected_tools"])
cancel_count = sum(1 for c in CASE_SET if "cancel_booking" in c["expected_tools"])
print("casos que usan get_quote    :", quote_count)
print("casos que usan cancel_booking:", cancel_count)

casos que usan get_quote    : 4
casos que usan cancel_booking: 1

Exercise 2: Confirm the reset also clears a booking from an earlier case (Medium)

Book something manually with rt.book_room(...) (outside any CASE_SET case, simulating "prior work" in the same process). Confirm rt.BOOKINGS isn't empty. Call reset_reservo_state(). Confirm rt.BOOKINGS is empty again, and the next booking gets booking_id: 1 again.

rt.book_room(room="Studio", tier="basic", hours=1, member="Prueba previa")
print("BOOKINGS antes del reset:", rt.BOOKINGS)

reset_reservo_state()
print("BOOKINGS después del reset:", rt.BOOKINGS)

nueva = rt.book_room(room="Focus", tier="pro", hours=3, member="Ana")
print("nueva reserva tras el reset:", nueva)

BOOKINGS antes del reset: {1: {'booking_id': 1, 'room': 'Studio', 'tier': 'basic', 'hours': 1, 'member': 'Prueba previa', 'price_cents': 4000}}
BOOKINGS después del reset: {}
nueva reserva tras el reset: {'booking_id': 1, 'confirmed': True}

Exercise 3: Design check_case_shape, a form check for the CASE_SET itself (Hard)

Common mistake 5 names a real problem: a malformed case in golden_cases.json doesn't fail when it loads, only later, with a confusing error. Write a check_case_shape(case) function that confirms a case has the seven required fields (name, question, model_script, expected_tools, expected_output, cost_threshold_cents, latency_threshold_ms), returning a list of missing fields. Test it against a complete case from the real CASE_SET, and against an incomplete case you build by hand (without latency_threshold_ms).

REQUIRED_CASE_FIELDS = [
    "name", "question", "model_script", "expected_tools",
    "expected_output", "cost_threshold_cents", "latency_threshold_ms",
]


def check_case_shape(case):
    """Chequeo de FORMA sobre el CASE_SET mismo: confirma que cada caso
    declara los siete campos que el harness necesita, ANTES de correrlo."""
    return [field for field in REQUIRED_CASE_FIELDS if field not in case]


incomplete_case = {"name": "roto", "question": "x", "model_script": [], "expected_tools": [],
                    "expected_output": {}, "cost_threshold_cents": 5}  # falta latency_threshold_ms

print("caso real         :", check_case_shape(CASE_SET[0]))
print("caso incompleto    :", check_case_shape(incomplete_case))

caso real         : []
caso incompleto    : ['latency_threshold_ms']

Summary and next step

• We built regression/golden_cases.json: five fixed cases, each with its question, its turn script, the expected tool sequence, the exact expected result, and its cost and latency thresholds.
• We confirmed why "fixed" is the property that makes a regression gate reliable: if the CASE_SET doesn't change, a FAIL can only mean the system changed.
• We built reset_reservo_state, and confirmed, with a real case, that without it the order cases run in can produce a FAIL with nothing to do with any real regression in the agent.
• load_case_set closes this lesson's work: five cases, loaded as pure data, ready for lessons 04 through 07 to run them against lesson 02's four functions.

Next lesson: 04 — Form, Not Quality: the Boundary. With the CASE_SET already built, we dive deep into check_schema: the complete OUTPUT_SCHEMAS dictionary for Reservo's four tools, and this whole module's most important statement — which questions this check never answers, and why those questions belong to evaluation-frameworks-guide.

Additional resources

1. Python — json — json.load/json.dumps, the complete foundation of golden_cases.json and load_case_set.
2. Python — itertools.count — The counter reset_reservo_state rebuilds from scratch before every case, the same tool you already used for _booking_ids since agent-fundamentals M2.
3. JSON Lines — The format this module deliberately avoids for golden_cases.json: a CASE_SET is a single array, not a sequence of independent events, so standard JSON (not NDJSON) is the right choice here.
4. Anthropic — Building effective agents — On why a stable, repeatable test set is the foundation of any reliability discipline for an agentic system.
5. Python 3.14 — What's New — The version every line of code in this lesson ran on.