Go or No-Go

Description

Lessons 04 and 05 left the complete evidence: v1 passes PASS (5/5); v2 fails FAIL (4/5), exactly on quote_focus_pro_3h, and the check that caught it was tool_choice_ok, with a real side effect on latency_ok. This lesson turns that evidence into a decision: a function, rollout_decision, that takes two GateReports — the old version's, the new version's — and returns a single word, "GO" or "NO-GO", along with the exact list of cases that justify it.

The rule is deliberately strict, and this lesson runs it against two scenarios: v2 (which breaks it) and a hypothetical v3 that fixes the regression (which meets it). The contrast between the two results is the lesson's point: the same rule, applied unchanged, produces opposite decisions depending on the evidence.

Connection to the module

This lesson picks the registry (Lesson 03) and the comparison (Lesson 04) back up and gives them a purpose: a decision isn't useful if it stays as a GateReport someone has to read case by case from memory. rollout_decision is the piece that translates that evidence into a concrete action, with an explicit rule anyone can audit without having to trust the judgment of whoever is looking at the report.

The rule, in one sentence

From this guide's DISEÑO: v2 gets GO if it passes the gate as well as or better than v1; it gets NO-GO if it breaks even a single case v1 used to pass. Notice the nuance, because it's the rule's heart: it isn't "if the total count improves or stays the same," it's "if no case that used to pass now fails." The difference matters — a new version could fix a broken case and break another, ending with the same total count as before, and still deserve a NO-GO, because it introduced a real regression in a case that worked.

def rollout_decision(gate_old, gate_new):
    """GO si la version nueva pasa el gate igual o mejor que la vieja: NUNCA
    puede romper un caso que la vieja pasaba. NO-GO en caso contrario.
    Devuelve (decision, lista_de_nombres_de_casos_rotos)."""
    old_by_name = {c.name: c for c in gate_old.cases}
    new_by_name = {c.name: c for c in gate_new.cases}
    broken = [
        name for name, old_c in old_by_name.items()
        if old_c.passed and not new_by_name[name].passed
    ]
    return ("NO-GO", broken) if broken else ("GO", [])

broken is a list comprehension that walks only the cases gate_old.cases used to pass (if old_c.passed) and checks whether, in gate_new.cases, that same name stopped passing. A case gate_old already failed, and that gate_new still fails (or even fixes), never enters broken — it isn't a regression, it's, at worst, a problem that already existed before.

Running the decision: v1 against v2

decision, broken = rollout_decision(report_v1, report_v2)
print(f"v1: {'PASS' if report_v1.passed else 'FAIL'} ({sum(c.passed for c in report_v1.cases)}/{len(report_v1.cases)})")
print(f"v2: {'PASS' if report_v2.passed else 'FAIL'} ({sum(c.passed for c in report_v2.cases)}/{len(report_v2.cases)})")
print(f"rollout_decision(v1, v2) -> {decision}, casos_rotos={broken}")

What to expect:

v1: PASS (5/5)
v2: FAIL (4/5)
rollout_decision(v1, v2) -> NO-GO, casos_rotos=['quote_focus_pro_3h']

NO-GO, and the list ['quote_focus_pro_3h'] records exactly why — with no ambiguity, with no need to reread the complete report to understand the reason.

Running the decision: v1 against a v3 that fixes the regression

To see the rule produce the opposite result, imagine the team catches v2's problem and ships a v3 with the proactivity instruction fixed — more precise about when it applies, without affecting quote-only questions. Its behavior, for the CASE_SET's five cases, matches v1's again: VERSION_OVERRIDES["v3"] = {}, with no substitution at all.

VERSION_OVERRIDES["v3"] = {}

report_v3 = run_regression_gate(CASE_SET, overrides=VERSION_OVERRIDES["v3"])
decision_v3, broken_v3 = rollout_decision(report_v1, report_v3)
print(f"v3: {'PASS' if report_v3.passed else 'FAIL'} ({sum(c.passed for c in report_v3.cases)}/{len(report_v3.cases)})")
print(f"rollout_decision(v1, v3) -> {decision_v3}, casos_rotos={broken_v3}")

What to expect:

v3: PASS (5/5)
rollout_decision(v1, v3) -> GO, casos_rotos=[]

Same function, same thresholds, not a single line of rollout_decision changed between the two runs — the result changed because the evidence changed. That's, exactly, the property that makes an explicit rule trustworthy: there's no last-minute judgment call, no courtesy exception for a version that "almost" works.

Common mistakes

1. Comparing only the total count (4/5 versus 5/5) instead of the specific cases. Two versions with the same total count can have completely different sets of failures — one that fixes an old case and breaks a new one looks, in the count, the same as the previous version, but it introduced a real regression this lesson's rule does detect.

2. Thinking a NO-GO means the new version is "worse overall." Not necessarily — v2 could, in a real scenario, improve the experience on ambiguous questions no CASE_SET case covers. rollout_decision doesn't evaluate "better overall"; it evaluates, precisely, "did it break something that already worked?" Those are different questions, and this lesson's rule only answers the second.

3. Letting a new version "buy" permission to break one case by fixing another. That's, precisely, the nuance the "The rule, in one sentence" section warns about — and the reason broken gets calculated case by case, never as a difference of total counts.

4. Running rollout_decision with the arguments in the wrong order. The function assumes the first argument is the old version (the baseline) and the second is the new version (the one being evaluated). Swapping the order flips the comparison's entire meaning — a case the "new" (actually the old) used to pass and the "old" (actually the new) doesn't, would produce a reading exactly backward from the real one.

5. Treating GO/NO-GO as the process's end, with the decision recorded nowhere. A decision that isn't written down anywhere (Lesson 08's mini-project's AGENT_CHANGELOG.md) gets lost the moment the terminal session ends — and the next person wondering "why are we still on v1 if there's a v2?" has no way to know without repeating all of this lesson's work.

Exercises

Exercise 1: Calculate the decision by hand, without running code (Easy)

gate_old has three cases: case_a (PASS), case_b (PASS), case_c (FAIL). gate_new has the same three cases, all PASS. Without running rollout_decision, decide what it would return: GO or NO-GO? Then, confirm with code.

from regression.harness import CaseResult, GateReport

old = GateReport(cases=[
    CaseResult(name="case_a", passed=True),
    CaseResult(name="case_b", passed=True),
    CaseResult(name="case_c", passed=False),
], passed=False)
new = GateReport(cases=[
    CaseResult(name="case_a", passed=True),
    CaseResult(name="case_b", passed=True),
    CaseResult(name="case_c", passed=True),
], passed=True)
print(rollout_decision(old, new))

('GO', [])

Exercise 2: A version that fixes one case and breaks another is still NO-GO (Medium)

Build two GateReports with two cases each: in gate_old, quote_focus_basic_3h passes and book_and_cancel_studio_basic_1h_diego fails. In gate_new, quote_focus_basic_3h fails (a new regression) and book_and_cancel_studio_basic_1h_diego passes (it got fixed). Confirm that, even though the total count is the same in both (1/2), rollout_decision returns NO-GO.

old2 = GateReport(cases=[
    CaseResult(name="quote_focus_basic_3h", passed=True),
    CaseResult(name="book_and_cancel_studio_basic_1h_diego", passed=False),
], passed=False)
new2 = GateReport(cases=[
    CaseResult(name="quote_focus_basic_3h", passed=False),
    CaseResult(name="book_and_cancel_studio_basic_1h_diego", passed=True),
], passed=False)

print("gate_old:", sum(c.passed for c in old2.cases), "/", len(old2.cases))
print("gate_new:", sum(c.passed for c in new2.cases), "/", len(new2.cases))
decision2, broken2 = rollout_decision(old2, new2)
print("decision:", decision2, "broken:", broken2)

gate_old: 1 / 2
gate_new: 1 / 2
decision: NO-GO broken: ['quote_focus_basic_3h']

Exercise 3: Design a rule variant that DOES tolerate a trade-off (Hard)

A different team might deliberately decide a trade-off like Exercise 2's is acceptable if the total count improves or stays the same. Write rollout_decision_lenient(gate_old, gate_new) implementing that alternative policy (GO if the number of passing cases in gate_new is greater than or equal to gate_old's, regardless of which specific cases changed), test it on Exercise 2's same scenario, and explain in two sentences why this guide doesn't adopt that policy as the default rule.

def rollout_decision_lenient(gate_old, gate_new):
    """Politica alternativa: GO si el conteo total no empeora, sin
    importar que casos especificos cambiaron."""
    old_n = sum(c.passed for c in gate_old.cases)
    new_n = sum(c.passed for c in gate_new.cases)
    return ("GO", []) if new_n >= old_n else ("NO-GO", [])

decision_lenient, _ = rollout_decision_lenient(old2, new2)
print("rollout_decision_lenient:", decision_lenient)

rollout_decision_lenient: GO

Summary and next step

• rollout_decision(gate_old, gate_new) implements the DISEÑO's rule: GO if the new version never breaks a case the old one used to pass; NO-GO otherwise, with the exact list of broken case names as evidence.
• Run over v1 against v2: NO-GO, casos_rotos=['quote_focus_pro_3h'] — the same regression from Lessons 04 and 05, now turned into a formal decision.
• Run over v1 against a fixed v3 (PASS 5/5): GO, casos_rotos=[] — the same function, without changing a single line, produces the opposite result because the evidence changed.
• The rule is deliberately strict: it never lets an improvement in one case "buy" permission to break another — every case gets evaluated independently, with no averaging.

Next lesson: 07 — Rolling Back. With v2's NO-GO decision already made, we build rollback: how the system goes back, deterministically, to the previous version — and why, in this guide, that's a one-line pointer change.

Additional resources

1. Python — dictionary and list comprehensions — The foundation of broken = [... for ... if ...], rollout_decision's heart.
2. Anthropic — Building effective agents — On why an agent's success criteria must be explicit and verifiable, not subjective impressions.
3. Python — dataclasses and field comparison — The shape of CaseResult and GateReport rollout_decision walks.
4. Python 3.14 — What's New — The version every line of code in this lesson ran on.