When to stop: the same mechanism, two different results

Description

guardrailWatch() already gave you a verdict on recommendations' real rollout: HALT at the 10% stage. This lesson stays right there and asks what that word means in practice — and, to make it impossible to confuse, runs the exact same mechanism on a second, hypothetical scenario, where the engineering team has already fixed the latency problem before restarting the rollout. Comparing the two results — one that stops, one that reaches 100% — is the clearest way to see that guardrailWatch() has no opinion on whether recommendations is a good idea: it only reports, with discipline, whether each stage's data respects the limits the team set in advance.

Connection to the module. This lesson doesn't change a single line of guardrailWatch() or guardrailCheck() — it reuses them exactly as they stood in lesson 3; the only thing that changes is the dataset passed to them. That's, deliberately, this lesson's central point: the discipline of stopping doesn't depend on inventing a different function for each situation, it depends on running the same function honestly on the data that was actually measured.

An analogy: the temperature needle doesn't negotiate with speed

Going back to lesson 1's car dashboard: imagine the temperature needle enters the red zone while you're going at a good speed on a clear highway. A driver who reasons "I'm going fast, the road is clear, I'll go a bit further and check the temperature later" is making a dangerous decision disguised as patience. The temperature needle doesn't ask speed whether it can wait — when it enters the red zone, the correct action is the same no matter how good the dashboard's other numbers look: reduce speed, now, not "in a bit."

checkoutConversionRate climbing strongly is the clear highway. checkoutLatencyP95Ms at 910ms, over an 800 ceiling, is the needle in the red zone. The mistake this lesson names by its full name is reasoning "the primary metric is doing well, let's go a bit further before stopping" — exactly the same mistake as the driver in the analogy.

Worked example: the same guardrailWatch(), two different ramps

// guardrailCheck and guardrailWatch: EXACTLY the same as lesson 3, with no
// changes at all. The only thing that changes in this lesson is the dataset
// passed to them.
function guardrailCheck(before, after, guardrails) {
  const results = guardrails.map((g) => {
    const beforeVal = before[g.metric];
    const afterVal = after[g.metric];
    let broken = false;
    let detail = '';
    if (g.type === 'ceiling') {
      broken = afterVal > g.limit;
      detail = afterVal + ' vs ceiling ' + g.limit;
    } else if (g.type === 'floor') {
      broken = afterVal < g.limit;
      detail = afterVal + ' vs floor ' + g.limit;
    } else if (g.type === 'maxIncrease') {
      const delta = afterVal - beforeVal;
      broken = delta > g.limit;
      detail = 'delta +' + delta.toFixed(4) + ' vs max allowed +' + g.limit;
    }
    return { metric: g.metric, before: beforeVal, after: afterVal, broken, detail };
  });
  const brokenGuardrails = results.filter((r) => r.broken).map((r) => r.metric);
  return { results, anyBroken: brokenGuardrails.length > 0, brokenGuardrails };
}

function guardrailWatch(stages, guardrails, baseline) {
  const log = [];
  let halted = false;
  for (const stage of stages) {
    if (halted) {
      log.push({ stage: stage.name, percent: stage.percent, status: 'not reached (ramp halted earlier)' });
      continue;
    }
    if (!stage.metrics) {
      log.push({ stage: stage.name, percent: stage.percent, status: 'not measured yet' });
      continue;
    }
    const check = guardrailCheck(baseline, stage.metrics, guardrails);
    if (check.anyBroken) {
      log.push({ stage: stage.name, percent: stage.percent, status: 'HALT', broken: check.brokenGuardrails });
      halted = true;
    } else {
      log.push({ stage: stage.name, percent: stage.percent, status: 'continue' });
    }
  }
  return { log, halted };
}

const baseline = { checkoutLatencyP95Ms: 650, complaintRate: 0.012, churnRate: 0.045, grossMargin: 0.220 };
const guardrails = [
  { metric: 'checkoutLatencyP95Ms', type: 'ceiling', limit: 800 },
  { metric: 'complaintRate', type: 'maxIncrease', limit: 0.005 },
  { metric: 'churnRate', type: 'maxIncrease', limit: 0.010 },
  { metric: 'grossMargin', type: 'floor', limit: 0.180 },
];

// Hypothetical scenario: before restarting the rollout, engineering cached
// the call to the recommendations engine (the HOW of that technical fix is
// module 5's work; here we only use the hypothetical result after the fix,
// to contrast against lesson 3's real scenario).
const fixedStages = [
  { name: 'canary', percent: 1, metrics: { checkoutLatencyP95Ms: 670, complaintRate: 0.012, churnRate: 0.045, grossMargin: 0.220 } },
  { name: 'ramp-10', percent: 10, metrics: { checkoutLatencyP95Ms: 705, complaintRate: 0.012, churnRate: 0.045, grossMargin: 0.220 } },
  { name: 'ramp-50', percent: 50, metrics: { checkoutLatencyP95Ms: 740, complaintRate: 0.013, churnRate: 0.046, grossMargin: 0.219 } },
  { name: 'full', percent: 100, metrics: { checkoutLatencyP95Ms: 765, complaintRate: 0.013, churnRate: 0.046, grossMargin: 0.219 } },
];

console.log('=== guardrailWatch: same mechanism, hypothetical already-fixed scenario ===\n');
const watch = guardrailWatch(fixedStages, guardrails, baseline);
watch.log.forEach((e) => console.log(e.stage + ' (' + e.percent + '%): ' + e.status + (e.broken ? ' -- ' + e.broken.join(', ') : '')));
console.log('\nRamp halted: ' + watch.halted);

What to expect. Running the file with Node, the output is exactly this:

=== guardrailWatch: same mechanism, hypothetical already-fixed scenario ===

canary (1%): continue
ramp-10 (10%): continue
ramp-50 (50%): continue
full (100%): continue

Ramp halted: false

Compare this output against lesson 3's, line by line. There, latency climbed from 680ms (canary) to 910ms (10%), breaking the ceiling. Here, with the recommendations engine already cached (hypothetically), latency climbs much more slowly across the four stages — 670, 705, 740, 765 — and never crosses 800ms, not even at 100%. The result: Ramp halted: false. Not a single line of guardrailWatch() or guardrailCheck() changed between the two scenarios — the only thing different is the real data measured in production. That is, exactly, the property that makes this mechanism trustworthy: it doesn't decide in advance whether recommendations "deserves" to reach 100%, it decides, at every stage, whether the real data respects the limits — and two different rollouts of the same feature can have completely different verdicts, depending on how well the technical problem got fixed before retrying.

Going deeper: stopping isn't the same as canceling

This lesson's most expensive mistake isn't "continuing to climb when you shouldn't" — you already covered that in lesson 3. It's the opposite mistake, almost as costly: reading a HALT as if it meant "cancel recommendations forever." It doesn't. HALT at the 10% stage precisely says three things, and only three: the ramp doesn't advance to 50% yet; the users already at the 10% stage stay exposed to the problem while it gets decided what to do with them — that decision, revert or fix forward, belongs to module 5; and the technical diagnosis (why did latency climb so much between canary and 10%, not just a little?) has to get resolved before trying the ramp again. This lesson's hypothetical scenario — cached latency, full ramp with no HALT — is exactly that second attempt, after the technical problem got solved: same feature, same watching mechanism, different result because the root cause is no longer there.

Common mistakes

Watching only the primary metric and not the guardrails. What happens: seeing checkoutConversionRate keep climbing strongly throughout the rollout, someone argues that number, by itself, already justifies raising the percentage further — without having run guardrailWatch() at all, or having run it and deciding to ignore its result. Why it happens: the primary metric is the one everyone understands and the one reported up the organization; guardrails feel like a secondary, almost bureaucratic check, compared to the number that "really matters." How to spot it: if a decision to advance the ramp gets made with nobody mentioning checkoutLatencyP95Ms's, complaintRate's, churnRate's, or grossMargin's current state, the decision is incomplete. How to fix it: guardrailWatch() evaluates guardrails independently of the primary metric's value — by design, it doesn't even take checkoutConversionRate as input. That independence is what makes it possible for a feature to "win" and, at the same time, not be ready for 100%.

Continuing to advance the ramp even though a guardrail already broke, "because the primary metric is winning." What happens: the team sees the HALT at the 10% stage, acknowledges latency broke, and decides to climb to 50% anyway, arguing "18.75% lift in conversion outweighs 110ms of extra latency." Why it happens: converting two different metrics — conversion and latency — into a single "net score" feels like a way to reason quantitatively, even though it's actually comparing things the team, before launching, had already decided should not be compared against each other — that's why a guardrail exists with a fixed ceiling, instead of leaving the decision open for discussion in the moment. How to spot it: the phrase "the primary compensates for it" or "the trade-off is worth it" showing up after a HALT, with no diagnostic plan behind it. How to fix it: a guardrail with a fixed threshold, defined before launching, exists exactly so this negotiation doesn't happen "in the heat of the moment" — the moment to decide whether 110ms of extra latency is worth it was the planning meeting, not halfway through the rollout with the HALT already active.

Not defining the halt threshold in advance, and rationalizing it live. What happens: the team didn't set, before launching, what exactly "stopping" means — a single broken guardrail? two? only if the broken one is latency? — and ends up debating that definition in the middle of the rollout, with the pressure of a real decision on top of it. Why it happens: defining the halt criterion in advance requires thinking through scenarios that haven't happened yet, which feels less urgent than solving the problem that's already happening. How to spot it: if the question "does this count as a reason to stop?" gets answered differently depending on who's in the room that day, the criterion was never clearly set. How to fix it: guardrailWatch() uses the same rule as guardrailCheck() — anyBroken, a single broken guardrail is enough — defined in the code, not in an in-the-moment conversation; that rule gets decided before the first user sees recommendations, exactly as the metrics guide warned about the thresholds themselves.

Exercises

Exercise 1 — Fix the hypothetical scenario. If, in this lesson's "already fixed" scenario, the 50% stage had measured checkoutLatencyP95Ms: 810 instead of 740, would the final result change? At which stage would the ramp stop?

See solution

Yes, it would change. With 810 > 800, guardrailCheck()'s condition would flag that guardrail as broken at the ramp-50 stage, and guardrailWatch() would stop there with HALT — even though the two previous stages (canary at 670, ramp-10 at 705) had passed clean. The full (100%) stage would become not reached. This shows a technical fix can solve the problem at the early stages and still not be enough to hold the guardrail all the way to the end — the fix caches the call, but if the effect degrades with more concurrent load at 50%, the guardrail can break there instead of at 10%.

Exercise 2 — Argue against "the primary compensates for it." A teammate tells you: "I know latency broke, but the conversion lift is +18.75%, so it's worth putting up with a bit more latency while we keep climbing." Using this lesson's vocabulary, what would you answer, in 2-3 sentences?

See solution

A reasonable answer: "We didn't set the 800ms ceiling at random — we defined it before launching, precisely so we wouldn't have to negotiate in the middle of the rollout whether more latency is worth more conversion. If we now decide it is worth it, what we're doing is changing the rule after breaking it, not applying it. Let's follow the plan: we stop here, we diagnose why latency climbed so much, and if the technical fix holds the guardrail, we pick the ramp back up — that doesn't cancel the conversion lift, it just postpones capturing it until we can do it without the cost we already know exists."

Exercise 3 — Define the halt criterion for a new case. Mercado's logistics team is going to launch deliveryEtaV2 (the improved delivery time algorithm, already mentioned in previous modules) with its own ramp and its own guardrails. Before the rollout starts, write, in one sentence, the halt criterion they should set in advance — using guardrailWatch()'s same anyBroken rule, not a new one.

See solution

A reasonable criterion: "deliveryEtaV2's ramp stops, without exception, at the first stage where guardrailWatch() reports HALT for any of its defined guardrails — it doesn't need all of them to break, nor does the team need to agree in the moment that a single one already broke its threshold — and it doesn't resume until that broken guardrail's root cause has a diagnosis and, if applicable, a fix verified in a new run." The key to the exercise is noticing the criterion doesn't depend on which feature it is — recommendations or deliveryEtaV2 — because guardrailWatch() and its anyBroken rule are generic; the only thing that changes between features is the specific guardrails and their thresholds.

Summary and next step

In this lesson you ran the same guardrailWatch(), without changing a line, on two scenarios: the real one, where latency breaks the guardrail at 10% and the ramp stops; and a hypothetical one, where a prior technical fix keeps latency under control across all four stages and the ramp reaches 100% in full. The comparison leaves the central argument unambiguous: the mechanism has no opinion on whether recommendations is good or bad — it only applies, with discipline, the criterion the team set before launching. You also saw why HALT isn't the same as canceling, and why "the primary compensates for it" is, precisely, the reasoning a guardrail with a fixed threshold exists to prevent.

Before moving on you should be able to: explain the difference between stopping a ramp and canceling a feature; argue, with this lesson's vocabulary, against the idea that a strong primary metric "buys" permission to ignore a broken guardrail; and anticipate why setting the halt criterion before launching avoids a difficult negotiation in the middle of the rollout.

Lesson 5 takes this same result — HALT at 10%, with its full detail of which guardrail broke and why — and designs how it should look on a dashboard: what information needs to be visible at a glance so nobody has to run code by hand to know what stage the ramp is at, right now.

Resources

• Google SRE Book, Chapter 4, "Service Level Objectives" — sre.google/sre-book/service-level-objectives. The chapter that argues for why thresholds get set beforehand, as an explicit commitment, and don't get renegotiated based on the result already observed — this lesson's foundation.
• LaunchDarkly, "Introducing Guardrail Metrics: best-practice metrics for every release" — launchdarkly.com/blog/introducing-guardrail-metrics. How a real guarded rollout automatically pauses or reverts upon a detected regression, without waiting for someone to decide "in the heat of the moment" whether it's worth continuing.
• Ronny Kohavi, "Guardrail Metrics for A/B Tests" — linkedin.com/pulse/guardrail-metrics-ab-tests-ronny-kohavi. Kohavi insists a broken guardrail doesn't get "compensated for" by a positive primary metric — they're different questions, the same distinction this lesson argues for.