Guardrails in flight: watching every stage of the ramp

Description

The previous lesson gave you the criterion for reading a guardrail result based on how many people are exposed. This lesson builds the module's central piece: guardrailWatch(), the function that walks module 3's full ramp — canary 1% → 10% → 50% → 100% — and, at every stage, runs guardrailCheck() against the baseline and decides whether the ramp continues or stops. It's not a function that replaces anything you've already built: it reuses guardrailCheck() exactly as it stood in the metrics guide, and adds one single new idea — the sequence of stages, and the discipline of stopping at the first one that breaks a guardrail, without evaluating the next ones.

Connection to the module. This is the function lesson 1 promised, and the one lesson 8's project is going to reuse without changing a line — the same pattern you already saw in module 2, where isEnabled() was built once and reused in the final project. Everything that follows in this module — when to stop (lesson 4), what to show on the dashboard (lesson 5), how to alert (lesson 6), the error budget (lesson 7) — takes this function as already built.

An analogy: flight control, leg by leg

A long flight isn't supervised with a single check at takeoff and another at landing. Air traffic control, and the crew itself, check the plane's state in legs: cruising altitude reached, fuel within range, systems responding — and if something goes out of range at any leg, the response isn't "let's wait and see if it fixes itself at the next leg," it's addressing the problem at that leg, before continuing toward the next point on the route. Nobody keeps climbing to cruising altitude with a cabin pressure alarm sounding, "because we're almost at the next checkpoint."

guardrailWatch() is exactly that leg-by-leg discipline, applied to recommendations' rollout. Every ramp stage — canary, 10%, 50%, 100% — is a leg. If a guardrail breaks at one leg, the function doesn't keep evaluating the following legs as if nothing happened: it stops right there, with the same criterion as a pilot who doesn't keep climbing to cruise with an active alarm.

Worked example: guardrailWatch() on recommendations' real rollout

// guardrailCheck: reused, WITHOUT changes, from the metrics guide
// (product-metrics-and-experimentation-guide, module 4, lesson 6).
function guardrailCheck(before, after, guardrails) {
  const results = guardrails.map((g) => {
    const beforeVal = before[g.metric];
    const afterVal = after[g.metric];
    let broken = false;
    let detail = '';
    if (g.type === 'ceiling') {
      broken = afterVal > g.limit;
      detail = afterVal + ' vs ceiling ' + g.limit;
    } else if (g.type === 'floor') {
      broken = afterVal < g.limit;
      detail = afterVal + ' vs floor ' + g.limit;
    } else if (g.type === 'maxIncrease') {
      const delta = afterVal - beforeVal;
      broken = delta > g.limit;
      detail = 'delta +' + delta.toFixed(4) + ' vs max allowed +' + g.limit;
    }
    return { metric: g.metric, before: beforeVal, after: afterVal, broken, detail };
  });
  const brokenGuardrails = results.filter((r) => r.broken).map((r) => r.metric);
  return { results, anyBroken: brokenGuardrails.length > 0, brokenGuardrails };
}

// guardrailWatch: NEW in this module. Walks the ramp's stages (M3) IN ORDER
// and, at each one, runs guardrailCheck() against the baseline. As soon as
// ONE stage breaks a guardrail, it marks HALT and stops watching -- the
// following ramp stages are never reached, and get reported as such, instead
// of being evaluated with data that, in practice, was never measured.
function guardrailWatch(stages, guardrails, baseline) {
  const log = [];
  let halted = false;
  for (const stage of stages) {
    if (halted) {
      log.push({ stage: stage.name, percent: stage.percent, status: 'not reached (ramp halted earlier)' });
      continue;
    }
    if (!stage.metrics) {
      log.push({ stage: stage.name, percent: stage.percent, status: 'not measured yet' });
      continue;
    }
    const check = guardrailCheck(baseline, stage.metrics, guardrails);
    if (check.anyBroken) {
      log.push({ stage: stage.name, percent: stage.percent, status: 'HALT', broken: check.brokenGuardrails, results: check.results });
      halted = true;
    } else {
      log.push({ stage: stage.name, percent: stage.percent, status: 'continue', results: check.results });
    }
  }
  return { log, halted };
}

// The case: recommendations' rollout at Mercado, watched stage by stage. The
// baseline and the four guardrails are EXACTLY the ones from the metrics
// guide -- nothing gets redefined here.
const baseline = {
  checkoutLatencyP95Ms: 650,
  complaintRate: 0.012,
  churnRate: 0.045,
  grossMargin: 0.220,
};

const guardrails = [
  { metric: 'checkoutLatencyP95Ms', type: 'ceiling', limit: 800 },
  { metric: 'complaintRate', type: 'maxIncrease', limit: 0.005 },
  { metric: 'churnRate', type: 'maxIncrease', limit: 0.010 },
  { metric: 'grossMargin', type: 'floor', limit: 0.180 },
];

// Module 3's ramp, with the metrics measured at each stage that WAS reached.
// The 50% and 100% stages don't have metrics yet because the rollout, in
// practice, never got there (that's precisely what this function is going to
// confirm).
const rampStages = [
  { name: 'canary', percent: 1, metrics: { checkoutLatencyP95Ms: 680, complaintRate: 0.012, churnRate: 0.045, grossMargin: 0.220 } },
  { name: 'ramp-10', percent: 10, metrics: { checkoutLatencyP95Ms: 910, complaintRate: 0.013, churnRate: 0.045, grossMargin: 0.219 } },
  { name: 'ramp-50', percent: 50, metrics: null },
  { name: 'full', percent: 100, metrics: null },
];

console.log('=== guardrailWatch: watching recommendations\' rollout, stage by stage ===\n');
const watch = guardrailWatch(rampStages, guardrails, baseline);
watch.log.forEach((entry) => {
  console.log(entry.stage + ' (' + entry.percent + '%): ' + entry.status);
  if (entry.results) {
    entry.results.forEach((r) => {
      console.log('    ' + r.metric.padEnd(22) + String(r.before).padStart(8) + ' -> ' + String(r.after).padStart(8) +
        '  ' + (r.broken ? 'BROKEN' : 'OK    ') + '  (' + r.detail + ')');
    });
  }
  if (entry.broken) console.log('    broken guardrail(s): ' + entry.broken.join(', '));
});
console.log('\nRamp halted: ' + watch.halted);

What to expect. Running the file with Node, the output is exactly this:

=== guardrailWatch: watching recommendations' rollout, stage by stage ===

canary (1%): continue
    checkoutLatencyP95Ms       650 ->      680  OK      (680 vs ceiling 800)
    complaintRate            0.012 ->    0.012  OK      (delta +0.0000 vs max allowed +0.005)
    churnRate                0.045 ->    0.045  OK      (delta +0.0000 vs max allowed +0.01)
    grossMargin               0.22 ->     0.22  OK      (0.22 vs floor 0.18)
ramp-10 (10%): HALT
    checkoutLatencyP95Ms       650 ->      910  BROKEN  (910 vs ceiling 800)
    complaintRate            0.012 ->    0.013  OK      (delta +0.0010 vs max allowed +0.005)
    churnRate                0.045 ->    0.045  OK      (delta +0.0000 vs max allowed +0.01)
    grossMargin               0.22 ->    0.219  OK      (0.219 vs floor 0.18)
    broken guardrail(s): checkoutLatencyP95Ms
ramp-50 (50%): not reached (ramp halted earlier)
full (100%): not reached (ramp halted earlier)

Ramp halted: true

Read the result in order, because order is exactly what matters. At canary (1%, module 1's 125 buyers), all four guardrails come out OK — and, as lesson 2 confirmed, checkoutLatencyP95Ms already has trustworthy signal at this stage (680ms, still far from the 800 ceiling), so this OK really is a real confirmation, not just absent evidence. The ramp advances to 10%. There, checkoutLatencyP95Ms jumps to 910ms — the exact same number the metrics guide had already measured in its final report — crosses the 800 ceiling, and guardrailWatch() marks that stage HALT. From there, halted stays true, and the 50% and 100% stages show up as not reached: the function doesn't make up data for those stages or pretend it evaluated them — it precisely reports that the ramp never got there.

Notice something important: of the four guardrails, only one breaks (checkoutLatencyP95Ms); the other three (complaintRate, churnRate, grossMargin) stay OK even at the stage where the HALT triggers. guardrailWatch() doesn't need all the guardrails to break to stop the ramp — one breaking is enough. It's the same rule you already saw in the metrics guide's guardrailCheck(): anyBroken is sufficient, unanimity isn't needed.

Going deeper: why the break matters as much as the if

It's worth looking closely at the function's mechanics, because the part that does the real work isn't the check.anyBroken condition — it's what happens after: the halted variable gets set to true, and from that moment on, every subsequent stage in the for loop goes straight into the loop's first if, without running guardrailCheck() even once more. That's not a minor implementation detail: it's, literally, the difference between a watching system that respects its own alarm and one that ignores it. If guardrailWatch() kept evaluating the 50% and 100% stages with hypothetical data after a HALT, it would be simulating a rollout that should never have advanced that far — exactly the mistake lesson 4 is going to name by its full name: continuing to climb the ramp "because the primary metric is winning," ignoring that a guardrail already gave the signal to stop.

Common mistakes

Evaluating every ramp stage at once, without respecting the order. What happens: someone runs guardrailCheck() directly on the 10% stage's data (or any stage) without having first confirmed the previous stages passed clean. Why it happens: if you already have every stage's data saved, it's tempting to review it in any order, as if they were independent rows in a table. How to spot it: if the guardrail report doesn't say, anywhere, "this stopped at stage X and didn't continue," the watching lost the sequence that makes it useful. How to fix it: guardrailWatch() processes stages in ramp order, and stops as soon as it finds a HALT — order is the part that turns a list of results into a decision of "how far we got."

Simulating data for the "not reached" stages to fill out the report. What happens: someone, uncomfortable seeing empty stages in the report, fills in ramp-50 and full with estimated or projected values, "so the dashboard looks complete." Why it happens: a report with gaps feels incomplete, and there's a real temptation to fill it with something, even if that something was never actually measured. How to spot it: if any number in the guardrail report doesn't correspond to a real measurement, but to a projection disguised as data, the report stops being trustworthy. How to fix it: not reached is, in itself, valuable information — it precisely says "the ramp never got here, and that was a decision, not an accidental gap." There's no need — and no benefit — to filling it with anything else.

Confusing "a single broken guardrail" with "we need to check whether the primary metric compensates." What happens: seeing the HALT at 10%, someone proposes continuing to 50% anyway "because checkoutConversionRate is still climbing strongly, and that outweighs it." Why it happens: when the primary number looks good, it's tempting to treat a broken guardrail as an acceptable cost in exchange for that gain, instead of a signal to stop. How to spot it: if the conversation after a HALT includes the phrase "but the primary metric is winning," with no mention of diagnosing the broken guardrail's cause first. How to fix it: guardrailWatch(), deliberately, doesn't receive checkoutConversionRate's value as input — the guardrail is evaluated on its own, with no way for the primary result to "buy" permission to ignore it. That separation is the discipline lesson 4 is going to develop in depth.

Exercises

Exercise 1 — Change the threshold and run it again. If checkoutLatencyP95Ms's ceiling had been 950ms instead of 800ms (the same data: canary 680, 10% at 910), would guardrailWatch() still mark HALT at the 10% stage? How far would the ramp get?

See solution

It wouldn't break, and the ramp would reach the end. With limit: 950, guardrailCheck()'s condition evaluates 910 > 950, which is false — the guardrail would pass OK at the 10% stage. Since the other three guardrails already came out OK in the original data, and there are no further stages with measured data (ramp-50 and full are still metrics: null in this example), the result would be continue at canary and at 10%, and not measured yet at the remaining two stages — no HALT anywhere. This exercise confirms, once again, that the result depends entirely on where the threshold was set: 800ms stops the ramp; 950ms would have let it advance with the same real data.

Exercise 2 — Add a fifth stage. Mercado's team wants to add an intermediate stage between canary and 10%, called 'ramp-5' with percent: 5, with measured metrics of checkoutLatencyP95Ms: 790 (the other three guardrails unchanged from canary). Where would the ramp stop now, and why?

See solution

With checkoutLatencyP95Ms: 790, the condition evaluates 790 > 800, which is false — the ramp-5 stage would pass as continue, still under the ceiling, though already very close (10ms of margin). The ramp would keep going to ramp-10, where 910 > 800 does break the guardrail, and it would stop there exactly as in the original example — just with an extra intermediate stage now showing latency already climbing progressively (650 → 790 → 910) before crossing the ceiling. This exercise is a good preview of lesson 7: that margin of barely 10ms at ramp-5 is, literally, an almost-exhausted error budget.

Exercise 3 — Explain "not reached" to someone outside the technical team. In 2-3 sentences, without using the word "HALT" or "guardrail," explain to someone on Mercado's commercial team why guardrailWatch()'s report says "not reached" for the 50% and 100% stages, instead of showing some number there.

See solution

A reasonable message: "When we reached 10% of exposed users, we noticed the page was taking longer than acceptable to load, so we decided not to keep raising the percentage until we fixed that. That's why the report has no numbers for 50% or 100% — it's not that we forgot to measure, it's that, on purpose, we haven't yet exposed that many people to the problem we already detected. Once we fix the slowness, we're going to pick the rollout back up from there, not from zero."

Summary and next step

In this lesson you built and ran guardrailWatch(), the module's central function: it walks module 3's ramp in order, reuses guardrailCheck() unchanged at every stage, and stops at the first one that breaks a guardrail. On recommendations' real rollout, the result was precise: canary passes clean (680ms, within the 800 ceiling), 10% breaks the latency guardrail (910ms) and triggers HALT, and the 50% and 100% stages come back marked as never reached — the ramp stopped exactly where it should, without advancing one step past the problem.

Before moving on you should be able to: explain why guardrailWatch() stops evaluating stages after a HALT, instead of continuing to check the rest of the ramp; identify, in the result, which of the four guardrails broke and why the other three aren't enough to "save" the decision; and anticipate what would happen with a different threshold without having to run the code again.

Lesson 4 stays right at this result — HALT at 10% — and asks what that word means in practice: what would happen to Mercado's user base if the team decided to ignore it and keep climbing anyway, and what explicit criterion should exist, in advance, so nobody has to decide "in the heat of the moment" whether a HALT gets respected or not.

Resources

• Google SRE Book, Chapter 6, "Monitoring Distributed Systems" — sre.google/sre-book/monitoring-distributed-systems. The formal reference on why a production system needs to be checked in legs, with the same four signals, consistently — the discipline guardrailWatch() automates.
• LaunchDarkly, "Introducing Guardrail Metrics: best-practice metrics for every release" — launchdarkly.com/blog/introducing-guardrail-metrics. How a real guarded-rollout platform automatically pauses a release when it detects a regression, the same behavior guardrailWatch() implements by hand.
• Ronny Kohavi, "Guardrail Metrics for A/B Tests" — linkedin.com/pulse/guardrail-metrics-ab-tests-ronny-kohavi. Kohavi's article on guardrails, the conceptual foundation this lesson carries from "a single check" to "continuous watching across a ramp."