Watching the ramp live: the HALT at 10%

Description

The previous lesson left the flag turned on at the canary (1%, 720ms, within the ceiling) and the full ramp designed through 100%. Now comes what no ramp design, however good, can replace: watching live what actually happens as traffic climbs. This lesson runs guardrailWatch() on the full ramp, with the metrics guide's four business guardrails — not just latency — and confirms exactly where and why it stops.

Connection to the module. This lesson reuses guardrailCheck(), guardrailWatch(), and errorBudgetTracker() exactly as they stood in module 4's lessons 3 and 7, with no change. The difference from this same capstone's previous lesson isn't the mechanism — it's the scope: where lesson 2 evaluated only p95Latency at the canary stage with rolloutPlan() (module 3's simple pattern), this lesson evaluates all four guardrails at once — latency, complaints, churn, and margin — at every stage with real data, with module 4's fuller pattern. This lesson stops exactly at the moment the HALT is confirmed; what to do with that confirmation is lesson 4's job, which follows this one.

An analogy: the co-pilot who reads the instruments at every leg, not just at takeoff

A successful takeoff doesn't guarantee a fully safe flight — it's barely the first leg. A trained co-pilot doesn't check the instruments once, at the start, and then trust everything stays fine; they check them at every leg of the flight, with the same attention, because conditions change with altitude, with weather, with elapsed time. An engine that sounds perfect on the runway can behave differently at ten thousand meters, under more load and more continuous use.

The previous lesson's canary was the takeoff: 720ms, within the ceiling, all in order. But the canary ran on barely 1% of the base, for a short time — very different conditions from what the system faces when ten times more concurrent traffic asks the same engine for recommendations, at the same time. This lesson is the co-pilot checking the instruments at the flight's next leg — the 10% stage — with the same attention as at takeoff, not less.

Worked example: guardrailWatch() on the full ramp

// M8 L03: watches recommendations's full ramp with the metrics guide's FOUR
// guardrails (not just latency), stage by stage, until confirming where and
// why it stops. guardrailCheck(), guardrailWatch(), and errorBudgetTracker()
// are EXACTLY module 4's (L3, L7), with no change.

function guardrailCheck(before, after, guardrails) {
  const results = guardrails.map((g) => {
    const beforeVal = before[g.metric];
    const afterVal = after[g.metric];
    let broken = false;
    let detail = '';
    if (g.type === 'ceiling') {
      broken = afterVal > g.limit;
      detail = afterVal + ' vs ceiling ' + g.limit;
    } else if (g.type === 'floor') {
      broken = afterVal < g.limit;
      detail = afterVal + ' vs floor ' + g.limit;
    } else if (g.type === 'maxIncrease') {
      const delta = afterVal - beforeVal;
      broken = delta > g.limit;
      detail = 'delta +' + delta.toFixed(4) + ' vs max allowed +' + g.limit;
    }
    return { metric: g.metric, before: beforeVal, after: afterVal, broken, detail };
  });
  const brokenGuardrails = results.filter((r) => r.broken).map((r) => r.metric);
  return { results, anyBroken: brokenGuardrails.length > 0, brokenGuardrails };
}
function guardrailWatch(stages, guardrails, baseline) {
  const log = [];
  let halted = false;
  for (const stage of stages) {
    if (halted) {
      log.push({ stage: stage.name, percent: stage.percent, status: 'not reached (ramp halted earlier)' });
      continue;
    }
    if (!stage.metrics) {
      log.push({ stage: stage.name, percent: stage.percent, status: 'not measured yet' });
      continue;
    }
    const check = guardrailCheck(baseline, stage.metrics, guardrails);
    if (check.anyBroken) {
      log.push({ stage: stage.name, percent: stage.percent, status: 'HALT', broken: check.brokenGuardrails, results: check.results });
      halted = true;
    } else {
      log.push({ stage: stage.name, percent: stage.percent, status: 'continue', results: check.results });
    }
  }
  return { log, halted };
}
function errorBudgetTracker(baselineValue, ceiling, stages) {
  const totalBudgetMs = ceiling - baselineValue;
  return stages.map((s) => {
    const consumedMs = s.value - baselineValue;
    const remainingMs = totalBudgetMs - consumedMs;
    const pctConsumed = (consumedMs / totalBudgetMs) * 100;
    return {
      stage: s.name, percent: s.percent, value: s.value, consumedMs, remainingMs, pctConsumed,
      status: remainingMs < 0 ? 'EXHAUSTED' : 'within budget',
    };
  });
}

console.log('=== Part 1: guardrailWatch on the full ramp ===\n');
const baseline = { checkoutLatencyP95Ms: 650, complaintRate: 0.012, churnRate: 0.045, grossMargin: 0.220 };
const guardrails = [
  { metric: 'checkoutLatencyP95Ms', type: 'ceiling', limit: 800 },
  { metric: 'complaintRate', type: 'maxIncrease', limit: 0.005 },
  { metric: 'churnRate', type: 'maxIncrease', limit: 0.010 },
  { metric: 'grossMargin', type: 'floor', limit: 0.180 },
];
const rampStages = [
  { name: 'canary', percent: 1, metrics: { checkoutLatencyP95Ms: 680, complaintRate: 0.012, churnRate: 0.045, grossMargin: 0.220 } },
  { name: 'ramp-10', percent: 10, metrics: { checkoutLatencyP95Ms: 910, complaintRate: 0.013, churnRate: 0.045, grossMargin: 0.219 } },
  { name: 'ramp-50', percent: 50, metrics: null },
  { name: 'full', percent: 100, metrics: null },
];
const watch = guardrailWatch(rampStages, guardrails, baseline);
watch.log.forEach((entry) => console.log(entry.stage + ' (' + entry.percent + '%): ' + entry.status));

console.log('\n=== Part 2: the responsible guardrail, with its detail ===\n');
const haltEntry = watch.log.find((e) => e.status === 'HALT');
const brokenDetail = haltEntry.results.find((r) => r.broken);
console.log('HALT stage: ' + haltEntry.stage + ' (' + haltEntry.percent + '%)');
console.log('Broken guardrail: ' + brokenDetail.metric + '  ' + brokenDetail.before + ' -> ' + brokenDetail.after + '  (' + brokenDetail.detail + ')');

console.log('\n=== Part 3: error budget consumed at that stage ===\n');
const budget = errorBudgetTracker(650, 800, [{ name: haltEntry.stage, percent: haltEntry.percent, value: brokenDetail.after }]);
const b = budget[0];
console.log(b.stage + ' (' + b.percent + '%): consumed=' + b.consumedMs + 'ms (' + b.pctConsumed.toFixed(1) +
  '% of the 150ms budget) | remaining=' + b.remainingMs + 'ms | ' + b.status);

What to expect. Running the file with Node, the output is exactly this:

=== Part 1: guardrailWatch on the full ramp ===

canary (1%): continue
ramp-10 (10%): HALT
ramp-50 (50%): not reached (ramp halted earlier)
full (100%): not reached (ramp halted earlier)

=== Part 2: the responsible guardrail, with its detail ===

HALT stage: ramp-10 (10%)
Broken guardrail: checkoutLatencyP95Ms  650 -> 910  (910 vs ceiling 800)

=== Part 3: error budget consumed at that stage ===

ramp-10 (10%): consumed=260ms (173.3% of the 150ms budget) | remaining=-110ms | EXHAUSTED

Notice something this module's lesson 2, with its single-criterion rolloutPlan(), couldn't show: at the canary, all four guardrails get checked — not just latency — and all four pass (continue). At ramp-10, it's specifically checkoutLatencyP95Ms that breaks — the other three (complaintRate, churnRate, grossMargin) stay within range — and that matters: the HALT isn't "everything is wrong," it's "one specific guardrail, with one specific number, crossed its limit." The ramp-50 and full stages never get reached, exactly as designed: guardrailWatch() doesn't keep evaluating stages after a HALT, no matter that their data was already defined in the code (here, in fact, it doesn't even exist — it's null, because it never got measured).

Part 3 translates the same finding into a magnitude: 260ms consumed against a 150ms budget, a 173.3% overrun, with a -110ms negative balance. That number — not just the word "broken" — is what separates a fine-tuning adjustment from a problem that needs a redesign: a guardrail exceeded by 5% calls for a patch; one exceeded by 173%, like this case, calls for investigating the root cause before climbing further.

Why this lesson uses guardrailWatch() and not rolloutPlan()

This module's lesson 2 used rolloutPlan(), with a single criterion (p95Latency <= 800), to confirm the canary advanced. This lesson uses guardrailWatch(), with four guardrails at once. The difference isn't cosmetic — it reflects two different questions a real launch needs to answer at different moments. rolloutPlan(), with its simple advanceIf(), is the right tool when you already know exactly which criterion matters and just need to decide whether to advance or not. guardrailWatch(), with its full guardrail list, is the right tool when you need to watch everything that could break, not just what you already suspect will break — exactly a real rollout's situation, where a latency problem could, in theory, show up alongside a rise in complaints or a margin drop, and the team needs to know which of those, if any, is actually happening.

In recommendations's case, it turns out only latency breaks — but the team didn't know that in advance. Watching all four guardrails and confirming three stay healthy is, in itself, valuable information: it rules out the hypothesis that the problem is broader than a specific technical bottleneck.

Common mistakes

Checking only the guardrail already suspected to be broken, ignoring the other three. What happens: someone, knowing in advance (from the metrics guide) that latency is the problem, runs guardrailCheck() only on checkoutLatencyP95Ms, without including complaintRate, churnRate, or grossMargin in the guardrails list. Why it happens: the expected result is already known, and checking the other three feels like unnecessary work when "we already know what the problem is." How to spot it: if your code's guardrails array has fewer than four elements, you aren't watching everything the metrics guide identified as relevant for this launch. How to fix it: as in this lesson's Part 1, the full guardrail list always gets watched, even when you already suspect which one is going to break — confirming the other three stay healthy is, itself, part of the report.

Confusing ramp-10's HALT with a canary failure. What happens: someone, seeing the result, concludes "the canary also failed," blending the two stages into a single conclusion. Why it happens: both stages belong to the same ramp, and it's easy to generalize a later stage's result backward. How to spot it: if your result summary doesn't explicitly distinguish between canary: continue and ramp-10: HALT, the problem's exact stage got lost in communication. How to fix it: as Part 1 shows, each stage has its own status — the canary passed cleanly with all four guardrails healthy; the problem showed up specifically once traffic climbed to 10%, not before. That distinction matters because it points to the cause: something related to volume or concurrency, not the feature itself at any scale.

Reporting the HALT without the error budget, leaving the problem's magnitude unquantified. What happens: someone communicates "latency broke at 10%" without adding Part 3's data — how much the budget was exceeded by — leaving the problem's real severity ambiguous. Why it happens: "broken" already sounds like enough information, and calculating the error budget seems like an optional extra step. How to spot it: if your report can't answer "by how much was it exceeded?", the magnitude that separates a minor adjustment from a serious problem is missing. How to fix it: as in this lesson's Part 3, always pair a HALT with the consumed error budget — a 173.3% overrun immediately communicates that this isn't a debatable edge case.

Exercises

Exercise 1 — What would have happened if only latency had gotten slightly worse? Suppose that, at ramp-10, checkoutLatencyP95Ms had measured 790ms instead of 910ms (still within the 800ms ceiling), with the rest of the data unchanged. Mentally run guardrailWatch() with that change. Does the ramp stop anywhere? What error budget would Part 3 show for that stage?

See solution

With checkoutLatencyP95Ms: 790, the condition afterVal > g.limit (790 > 800) is false — the latency guardrail doesn't break, and since the other three guardrails were already healthy in the original data, ramp-10 would go to continue. The ramp would keep evaluating ramp-50 and full — but since those two still have metrics: null in this exercise, guardrailWatch() would flag them as not measured yet (neither HALT nor NOT_REACHED), waiting for real data before deciding. The error budget at ramp-10 would be consumedMs: 790 - 650 = 140, against a 150 budget — 93.3% consumed, remainingMs: 10, status: 'within budget' — within the limit, but with very little margin left, a signal it would be worth watching closely before trusting the next stage holds up too.

Exercise 2 — Apply the pattern to a different guardrail. Mercado's logistics team watches its delivery-time algorithm with an errorRate guardrail (type ceiling, limit 0.04) alongside the same three business guardrails (complaintRate, churnRate, grossMargin, unchanged). At its 10% stage, they measure errorRate: 0.038; the other three guardrails stay equal to the baseline. Does guardrailWatch() report continue or HALT at that stage?

See solution

continue. With type: 'ceiling' and limit: 0.04, the break condition is afterVal > g.limit, that is, 0.038 > 0.04, which is false — the guardrail doesn't break, even though it's close to the limit. With the other three guardrails also healthy (equal to the baseline, with no increase exceeding its maxIncrease or dropping below its floor), check.anyBroken would be false for that stage, and guardrailWatch() would continue to the next one. It's worth noting "close to the limit but not breaking it" (0.038 against 0.04) is a reasonable warning sign for the team, even though the code, correctly, doesn't treat it as a HALT — that distinction between "close" and "broken" is precisely what having an explicit numeric limit is for, instead of an imprecise judgment call.

Exercise 3 — Communicate the HALT with all three parts together. Write the message (80-120 words) you'd send to Mercado's team's incident channel at the exact moment guardrailWatch() confirms the HALT at ramp-10. Include: the stage, the responsible guardrail with its three numbers (before, after, ceiling), and the consumed error budget.

See solution

One possible message: "🔴 HALT confirmed — recommendations rollout, ramp-10 stage (10% of the base). checkoutLatencyP95Ms went from 650ms to 910ms, above the 800ms ceiling we set before launching. The other three guardrails (complaints, churn, margin) stay healthy. Error budget: 260ms consumed against a 150ms budget — 173.3% overrun, negative balance of -110ms. The ramp won't advance to 50% or 100% until this gets resolved. We need to decide in the next few minutes: do we revert or investigate with current exposure?" The message gives the exact stage, the guardrail's three numbers, the error budget's magnitude, and immediately raises the question the next lesson answers.

Summary and next step

In this lesson you watched recommendations's full ramp with guardrailWatch() over the four business guardrails: the canary passes cleanly (680ms, all four healthy), but ramp-10 stops with a HALT when checkoutLatencyP95Ms breaks the ceiling (910ms against 800ms), consuming 173.3% of the available error budget. The ramp-50 and full stages never get reached — exactly what the ramp is designed to do facing a broken guardrail.

Before moving on you should be able to: explain the difference between using rolloutPlan() (one criterion) and guardrailWatch() (several guardrails at once); and calculate by hand, given a checkoutLatencyP95Ms value, whether a stage would pass or break the 800ms ceiling.

Lesson 4 takes this confirmation — HALT at ramp-10, 25,000 buyers exposed, a broken critical guardrail — and answers the question exercise 3's message left open: revert, or fix forward? And, once decided, how fast did the team actually respond?

Resources

• Google SRE Book, Chapter 6, "Monitoring Distributed Systems" — sre.google/sre-book/monitoring-distributed-systems. The reference framework for watching systems with complete signals, not a single isolated metric — the basis for why this lesson uses four guardrails at once. In English.
• Google SRE Book, Chapter 3, "Embracing Risk" — sre.google/sre-book/embracing-risk. The chapter on error budgets behind this lesson's Part 3: how much margin was left, and by how much it got exceeded. In English.
• LaunchDarkly, "Introducing Guardrail Metrics: best-practice metrics for every release" — launchdarkly.com/blog/introducing-guardrail-metrics. How the industry automates, on real platforms, the discipline of watching several guardrails at once during a rollout. In English.