When a winner breaks a guardrail: the verdict on `recommendations`

Description

Six lessons built, piece by piece, this module's complete framework: the gap between "the experiment won" and "ready for everyone" (L2), the launch as a risk decision (L3), blast radius measured in number of people (L4), the deploy/release separation that makes controlling that radius possible (L5), and the three properties every careful launch shares (L6). This lesson, the module's last topic lesson, brings the six pieces together into a single verdict on the case that runs through the whole guide: recommendations won, and it breaks a guardrail. Do you launch it, or not?

Connection to the module. There's no new concept here — this lesson is, literally, applying the full framework from lessons 2 through 6 to the real data the metrics guide left on the table. It's a rehearsal of the reasoning you're going to repeat, with the full technical tools, in lesson 8's mini-project, and one you'll need again, with different mechanics, every time a statistical winner brings a known problem along with it.

An analogy: the doctor's verdict, with two results in hand

A patient shows up with two lab results in hand: one confirms the new treatment worked — the marker that needed to improve, improved, clearly and measurably. The other shows a real side effect, already present, not hypothetical: blood pressure rose more than expected. A doctor who only looks at the first result says "let's keep the full dose, it worked." A doctor who panics over the second says "let's stop the whole treatment." Neither one is thinking with the correct framework. The doctor who reasons correctly does a third thing: recognizes the treatment works — that's no longer in doubt — and designs a plan that keeps the benefit while containing and watching the side effect: maybe a lower dose at first, more frequent blood pressure checks, a clear plan of what to do if pressure keeps rising.

recommendations is exactly that patient. The first result (conversion +18.75%, p=0.0114) says it works. The second (p95 latency at 910ms, above the 800 ceiling) says there's a real side effect, already confirmed. This lesson's verdict isn't a black-and-white "yes" or "no" — it's the plan of the doctor who knows how to read both results at once.

Worked example: launchVerdict() on recommendations

Let's build the simplest possible decision model, with three boolean questions that summarize everything the previous lessons built: is the primary result significant? is a guardrail broken? and, if so, is there a way to contain the damage while deciding what to do?

// launchVerdict: the high-level decision (not the mechanics) when facing a
// winner that breaks a guardrail. "containable" refers to whether there's a
// way to expose only a fraction first -- the HOW (flags/rollout) is M2-M3.
function launchVerdict({ primarySignificant, guardrailBroken, containable }) {
  if (!primarySignificant) return 'do not launch: primary result is not even significant';
  if (!guardrailBroken) return 'launch with standard gradual rollout, no extra care needed';
  if (guardrailBroken && containable) return 'launch STARTING small, watching the guardrail before scaling';
  return 'do not launch yet: guardrail broken and no way to contain the damage';
}

const mercadoCase = {
  primarySignificant: true, // checkoutConversion lift +18.75%, p=0.0114
  guardrailBroken: true, // p95Latency 910ms > ceiling 800ms
  containable: true, // the option of a gradual rollout exists (M2-M3 give the how)
};

console.log('=== launchVerdict on recommendations at Mercado ===\n');
console.log('primarySignificant=' + mercadoCase.primarySignificant + '  guardrailBroken=' + mercadoCase.guardrailBroken + '  containable=' + mercadoCase.containable);
console.log('Verdict: ' + launchVerdict(mercadoCase));

const noContainmentCase = { ...mercadoCase, containable: false };
console.log('\nIf Mercado had NO way to expose only a fraction first:');
console.log('Verdict: ' + launchVerdict(noContainmentCase));

What to expect. Running the file with Node, the output is exactly this:

=== launchVerdict on recommendations at Mercado ===

primarySignificant=true  guardrailBroken=true  containable=true
Verdict: launch STARTING small, watching the guardrail before scaling

If Mercado had NO way to expose only a fraction first:
Verdict: do not launch yet: guardrail broken and no way to contain the damage

Notice the two outputs for the same case, with a single field different (containable). With containable: true — because the possibility of a gradual rollout exists, at least in principle — the verdict is to launch, carefully. With containable: false — imagine a world where Mercado had no infrastructure at all to expose a fraction, no feature flags, nothing — the verdict changes completely to "do not launch yet," even though the primary result remains exactly as significant. That comparison is this whole module's argument, condensed into two lines of output: the decision to launch doesn't depend only on whether the experiment won — it depends, just as strongly, on whether you have the capacity to contain what you already know is broken.

Why this isn't, yet, a complete launch plan

It's worth being honest about what launchVerdict() does not do. It doesn't decide the exact percentage to start at (1%? 5%?) — that's module 3. It doesn't decide which specific guardrail to watch at each stage, or how often — that's module 4. It doesn't decide what to do if, already in production, the guardrail gets worse instead of better — that's module 5. What launchVerdict() does do, and it's exactly module 1's job, is answer the highest-level question: does the path of "launching carefully" even exist, given what we know? For recommendations, the answer is yes — and that answer is what authorizes modules 2 through 5 to start building the how.

This is the exact boundary between this module and the rest of the guide: module 1 answers whether it's worth attempting a careful launch; modules 2 through 5 are, each, a different piece of how to execute it.

Common mistakes

Simplifying the case to "it won, so launch it" or "it broke a guardrail, so don't launch it." What happens: in a rushed conversation, someone collapses launchVerdict()'s three variables into a single one — "did it win or not?" — and makes the decision on that one question alone. Why it happens: handling three variables at once takes more mental effort than handling one, and under time pressure it's tempting to simplify. How to spot it: the decision gets announced without mentioning, even once, the word "guardrail" or "containment." How to fix it: always use launchVerdict()'s three explicit questions, in that order — significance, guardrail, containment — before announcing any launch decision.

Treating containable as a fixed property of the problem, instead of a team capability. What happens: someone assumes that if a guardrail is broken, there's automatically "no way to contain it," without considering whether the team has — or can quickly build — the gradual-exposure infrastructure. Why it happens: the technical problem (latency) feels fixed and inevitable, and it's easy to forget that the capacity to contain it depends on tools the team controls, not on the bug itself. How to spot it: the conversation about "is it containable?" never mentions whether feature flags exist, or whether there's a way to measure the guardrail in real time. How to fix it: remember that containable in this model is a question about organizational capability (do we have the tools from modules 2 through 4?), not about the bug's severity — two completely different questions.

Confusing "start small" with "postpone the launch indefinitely." What happens: the team accepts the verdict of "launch starting small, watching the guardrail," but in practice never sets a date or a concrete plan, and the initial 1% stays there for months with nobody deciding to advance. Why it happens: "starting small" feels like a safe way to postpone a bigger decision indefinitely, without the discomfort of saying "we're not going to launch" out loud. How to spot it: weeks go by since deciding "launch carefully" and the exposure percentage is still the same as day one, with no explicit criteria for when to advance. How to fix it: "starting small" is the first step of a plan with defined stages — not a permanent state; module 3 of this guide teaches exactly how to design those stages with clear criteria for advancing.

Exercises

Exercise 1 — Apply the model to a different case. Mercado's sellers team runs a sellerDashboard experiment that wins with clear statistical significance, breaking no known guardrail. Use launchVerdict() mentally: what values would you assign to the three variables, and what verdict do you get?

See solution

primarySignificant: true, guardrailBroken: false, containable never gets evaluated because the function returns at the second if. The verdict: 'launch with standard gradual rollout, no extra care needed'. This is the simplest of the four cases the model can produce — without a broken guardrail, you don't need the extra level of caution recommendations does, though a standard gradual rollout (module 3, without module 4's extra intensive watching) is still a good default practice.

Exercise 2 — The case where nothing is containable yet. Imagine Mercado, today, is just starting to build its first feature flag infrastructure (this guide's module 2) and doesn't have anything functional yet. What does it mean, in practical and not just theoretical terms, for containable to be false at this moment? What would need to happen for it to become true?

See solution

In practical terms, containable: false means that today, if Mercado wanted to launch recommendations to a small fraction of users, it would have no real mechanism to do so without a full new deploy for every percentage change — exactly the problem discussed in lesson 3 (reversible, but slowly) and in lesson 5 (without the deploy/release separation, there's no way to expose only a fraction). For containable to become true, the team needs to build — not just wish for — module 2's feature flag infrastructure: without that concrete technical piece, the word "containable" is just an intention, not a real capability. This connects this lesson's verdict directly to the technical work that starts in module 2.

Exercise 3 — Write the message to the executive team. Imagine you have to explain, in 3-4 sentences, to someone outside the technical team — who only knows that "the recommendations experiment won" — why recommendations isn't launching to 100% this very week, despite the good news of the result. Use this lesson's vocabulary (significance, guardrail, containment) translated into language that person can follow.

See solution

A reasonable message: "Recommendations worked better than expected — 18.75% more people completed their purchase, and we're confident it wasn't chance. But, at the same time, we discovered that checkout becomes noticeably slower for users who see the carousel, more than we consider acceptable. Instead of choosing between 'launch it to everyone now' or 'cancel it entirely,' we're going to launch it starting with a very small fraction of users, watching that slowdown closely, and we'll only raise the percentage once we confirm we can keep it under control. That way we capture the real improvement without risking our whole base's shopping experience all at once." Notice the message names both results — the good one and the problematic one — with equal weight, and explains the middle-ground plan without requiring the other person to know the technical terms "canary" or "feature flag" yet.

Summary and next step

In this lesson, the module's last topic lesson, you brought the five previous lessons together into a single executed verdict: launchVerdict() confirms that recommendations — significant, with a broken guardrail, but with the possibility of containing the damage — should be launched starting small and watching the guardrail, not launched to 100% at once nor canceled entirely. You saw, through the comparison between containable: true and containable: false, that this containment capability isn't a property of the bug but of the tools the team has available — and that without them, even a perfectly significant result doesn't authorize a launch.

Before moving on you should be able to: apply launchVerdict()'s three questions to a new case, with your own judgment about what counts as significant, broken guardrail, and containable; explain why "starting small" is the first step of a plan, not a final state; and communicate the full verdict to someone with no technical vocabulary, without losing either half of the result.

With this you close the module's complete mental model. Lesson 8, the mini-project, puts you in the exact position of Mercado's team: comparing, with lesson 4's blastRadius() model, the real blast radius of "launching to 100% at once" against "starting with a 1% canary," and formally deciding the launch approach — the last step before module 2 gives you the first technical tool, feature flags, to actually build it.

Resources

• Google SRE Workbook, Chapter 16, "Canarying Releases" — sre.google/workbook/canarying-releases. The chapter that formalizes the practice of launching first to a small slice and evaluating before continuing, exactly the verdict launchVerdict() recommends for recommendations.
• Google SRE Book, Chapter 3, "Embracing Risk" — sre.google/sre-book/embracing-risk. The chapter on error budgets: how a team decides, with an explicit number, how much "unreliability" risk it's willing to tolerate before stopping — the conceptual foundation behind deciding what counts as containable.