The ramp: canary 1% → 10% → 50% → 100%

Description

This lesson draws the full ramp that will accompany the rest of the module and runs, for the first time, rolloutPlan() — the central model of the entire gradual rollout. Given a set of stages, each with a percentage and an advance criterion, rolloutPlan() walks the ramp in order and decides, stage by stage, whether to keep climbing or stop. You're going to run it today on recommendations' real case, with the latency guardrail you already know from module 1 — and you're going to see, with numbers, exactly where it stops.

Connection to the module. This lesson builds the piece lessons 3 through 7 are going to refine and extend: lesson 3 focuses on the first stage (the canary) and its minimum size; lesson 4 formalizes what, concretely, the advance criterion rolloutPlan() uses here in still-simple form means; lesson 5 applies module 1's blastRadius() to these same four stages; lesson 7 adds the minimum waiting time before trusting each measurement. All of them pick back up the same ramp you define today.

An analogy: the canary in the coal mine

Before electronic gas sensors existed, miners went down into coal mines with a cage and a canary inside. The canary is much more sensitive than a human to gases like carbon monoxide: if the air started getting poisoned, the canary would stop singing and fall before a miner noticed anything. Seeing the fallen canary was the signal to get out immediately — not after finishing the shift, not "to see if it gets better": immediately. The canary didn't solve the gas problem. It gave warning in time, at the lowest possible cost, before the problem reached someone who actually mattered.

The canary release — the ramp's first stage, the 1% — plays exactly that role. It doesn't fix recommendations' latency guardrail. What it does is expose such a small slice of users that, if the problem shows up, the cost of having discovered it there is minimal — and the signal arrives much earlier than if you'd waited to see what happened with 100% of the base. The rest of the ramp — 10%, 50%, 100% — are stages where, if the canary already "sang fine," you raise the stakes with more confidence at each step.

Worked example: rolloutPlan() on recommendations' rollout

The full ramp has four stages. At each one, we measure the latency guardrail (p95Latency, ceiling 800ms, already defined in the metrics guide) and decide whether to advance:

// rolloutPlan: defines the canary -> 10% -> 50% -> 100% ramp and, given the
// guardrail MEASURED at each stage, decides whether to advance or stop. It
// stops at the first stage that doesn't meet its criterion -- the following
// stages don't even get evaluated.
// Teaching case: recommendations' rollout at Mercado, with the latency
// guardrail (p95, ceiling 800ms) known from the metrics guide.
function rolloutPlan(stages) {
  const results = [];
  let halted = false;
  for (const s of stages) {
    if (halted) {
      results.push({ ...s, decision: 'NOT_REACHED' });
      continue;
    }
    const decision = s.advanceIf(s.measured) ? 'ADVANCE' : 'HOLD';
    results.push({ ...s, decision });
    if (decision === 'HOLD') halted = true;
  }
  return results;
}

const ceiling = 800; // p95Latency ceiling in ms, defined in the metrics guide

const stages = [
  { percent: 0.01, label: 'canary 1%', measured: { p95Latency: 720 }, advanceIf: (m) => m.p95Latency <= ceiling },
  { percent: 0.10, label: 'rollout 10%', measured: { p95Latency: 910 }, advanceIf: (m) => m.p95Latency <= ceiling },
  { percent: 0.50, label: 'rollout 50%', measured: { p95Latency: null }, advanceIf: (m) => m.p95Latency <= ceiling },
  { percent: 1.00, label: 'rollout 100%', measured: { p95Latency: null }, advanceIf: (m) => m.p95Latency <= ceiling },
];

console.log('=== rolloutPlan: recommendations\' ramp at Mercado (p95 ceiling=' + ceiling + 'ms) ===\n');
const plan = rolloutPlan(stages);
plan.forEach((s) => {
  const measuredLabel = s.measured.p95Latency === null ? 'n/a' : s.measured.p95Latency + 'ms';
  console.log(s.label.padEnd(14) + 'p95Latency=' + String(measuredLabel).padStart(6) + '  -> ' + s.decision);
});

What to expect. Running the file with Node, the output is exactly this:

=== rolloutPlan: recommendations' ramp at Mercado (p95 ceiling=800ms) ===

canary 1%     p95Latency= 720ms  -> ADVANCE
rollout 10%   p95Latency= 910ms  -> HOLD
rollout 50%   p95Latency=   n/a  -> NOT_REACHED
rollout 100%  p95Latency=   n/a  -> NOT_REACHED

⚠️ The p95Latency values per stage (720ms at the canary, 910ms at 10%) are a teaching case: they model how a small-scale canary doesn't always reproduce, at the same magnitude, a regression that does appear with more concurrent traffic. The 910ms number deliberately reuses the same value the metrics guide measured in the original experiment — the same regression, now seen at a bigger ramp stage.

Notice the last two rows: rollout 50% and rollout 100% show up as NOT_REACHED, not as HOLD or any other verdict on their own performance. rolloutPlan() never got to evaluating them — it stopped at rollout 10% and stayed there. That's the model's most important property: the ramp isn't a list of four independent checks that all run at once; it's a sequence, where each stage only gets evaluated if the previous one advanced. The 1% canary passed clean (720ms, under the ceiling). The 10% didn't — and that's where recommendations' rollout stands, for now.

Why the ramp stops, instead of "waiting to see what happens"

Notice something deliberate in rolloutPlan()'s design: when a stage results in HOLD, the function doesn't try the next stage "to see if it gets better." It stops right there. This isn't a minor technical detail — it's the difference between a real ramp and a list of checkboxes ticked with no consequence. If rolloutPlan() kept evaluating rollout 50% and rollout 100% anyway, the result of those rows would mean nothing: those people were never actually exposed, because the correct decision, upon seeing 910ms at the previous stage, is to stop before climbing further.

This also explains why the ramp has four stages and not two (1% and 100%, direct). Every intermediate stage is a chance to detect the problem with fewer people exposed than the next one. If the ramp were only 1% → 100%, and the 1% canary had passed clean (as it actually did, at 720ms), the next step would have been exposing the entire base — with no 10% intermediate step, which is exactly where this rollout revealed the real problem. More stages isn't bureaucracy: it's more chances for the "canary" to warn before the problem reaches everyone.

Common mistakes

Jumping from canary to 100% with no intermediate stages, "because the canary already passed." What happens: seeing that the 1% canary gave 720ms — clean, under the ceiling — someone proposes going straight to 100%, skipping the 10% and the 50%. Why it happens: "the canary passed" feels like sufficient evidence, and every intermediate stage looks like an unnecessary delay if the first one already came out fine. How to spot it: the proposal on the table is "canary clean, let's turn it on for everyone" with no mention of any stage between 1% and 100%. How to fix it: as today's example shows, recommendations' problem did not show up in the canary — it only appeared at the 10%, with more concurrent traffic. Without that intermediate stage, the rollout would have gone straight to 100% with the guardrail broken and no prior warning at all.

Treating NOT_REACHED stages as if they were "validated" or "pending approval" in some positive sense. What happens: someone reads rolloutPlan()'s output table and assumes rollout 50% and rollout 100% are "queued up," ready to advance as soon as the previous stage's problem gets resolved. Why it happens: NOT_REACHED visually looks like a neutral state, and it's easy to read it as "just hasn't had its turn yet" instead of "the rollout, as it stands, never got there." How to spot it: people talk about those stages as if they had their own data ("let's see how the 50% does"), when measured.p95Latency in those rows is literally null. How to fix it: NOT_REACHED means zero information about that stage — neither good nor bad. There's nothing to "advance" there until the previous stage stops being HOLD.

Designing a ramp with a single intermediate stage between the canary and 100%, thinking "fewer steps is simpler." What happens: someone proposes 1% → 100% directly, or 1% → 50% → 100%, arguing fewer stages means a faster, easier-to-coordinate process. Why it happens: every intermediate stage means coordination — someone has to review the numbers and decide — and fewer stages looks, at first glance, like less work. How to spot it: comparing the proposed ramp against this module's four-stage one, the jump between consecutive stages is much bigger (for example, from 1% to 50% is a 49-percentage-point jump, versus 1% to 10%, a 9-point jump). How to fix it: this module's lesson 5 puts an exact number on why big jumps are dangerous — how many new people end up exposed at each transition. For now, it's enough to notice that every stage you remove from the ramp is one fewer chance for the canary to warn you before it's too late.

Exercises

Exercise 1 — Change a stage's result and re-draw the ramp. Suppose that, instead of 910ms, the rollout 10% stage had measured 780ms (under the 800ms ceiling). Without running the code, draw rolloutPlan()'s full table with this new data — which stages advance, which stop, and which stay as NOT_REACHED?

See solution

With p95Latency: 780 at rollout 10% (780 <= 800, meets the criterion), that row becomes ADVANCE. Since no earlier stage stopped, rolloutPlan() keeps evaluating: rollout 50% and rollout 100% still have measured.p95Latency: null in this example, so with the data exactly as written in the code, both would still show the result of s.advanceIf(s.measured) evaluated against null — and null <= 800 is true in JavaScript, so, as the example is written, they would advance too. This reveals something important that isn't a flaw in the model but a limitation of this exercise's data: rolloutPlan() is only as trustworthy as the measured data you give it. In a real rollout, every stage needs its own real measurement before being evaluated — never null — exactly the topic of lesson 4 (advance criteria) and lesson 7 (waiting for enough data before looking).

Exercise 2 — Design a ramp with more stages. Mercado's team, after seeing the problem show up right at the jump from 1% to 10%, proposes adding an intermediate stage: 1% → 5% → 10% → 50% → 100%. What does the team gain with this extra stage? What does it lose?

See solution

Gains: one more chance to detect the problem with fewer people exposed — if the latency regression starts showing up gradually with more concurrent traffic (as suggested by it appearing at 10% and not at 1%), it might already be visible at 5%, before reaching 10%. That reduces, once again, the discovery's blast radius. Loses: total rollout time — every new stage means one more review, and probably one more minimum waiting time (dwell time, lesson 7) before being able to advance. The decision of how many stages to use is, at bottom, a balance between launch speed and how fine-grained you want early detection to be — there's no universally "correct" number, it depends on how much each additional week of rollout costs the business versus how much an incident not caught in time would cost.

Exercise 3 — Explain the model in your own words. A teammate, without having read this lesson, asks: "why doesn't rolloutPlan() evaluate all four stages at once and tell me the result for all of them?" Write the answer you'd give them, in 2-3 sentences.

See solution

A possible answer: "Because the stages aren't independent — they're sequential. It doesn't make sense to ask 'is the 50% rollout doing okay?' if you never exposed 50% of the people, because you stopped earlier at 10%. rolloutPlan() evaluates one stage at a time, in order, and as soon as one results in HOLD, it stops evaluating the following ones — because, in reality, those following stages never even got to run. The NOT_REACHED result is exactly that: there's no data to evaluate, because the rollout never got there."

Summary and next step

In this lesson you ran rolloutPlan(), the module's central model, on recommendations' four ramp stages: the 1% canary advanced clean (720ms), but the 10% broke the latency ceiling (910ms) and the ramp stopped there — the 50% and 100% stayed as NOT_REACHED, with no data backing them. You saw why the ramp evaluates stage by stage, in sequence, and why stopping in time at an intermediate stage is exactly the point of having more than two rungs between the canary and 100%.

Before moving on you should be able to: explain in your own words the difference between HOLD and NOT_REACHED; and run rolloutPlan() by hand given a set of stages and their measurements.

Lesson 3 dwells on the ramp's first stage — the canary — and answers a question this lesson left open: why start with such a small percentage, and what happens when that percentage is too small for the guardrail to give any trustworthy signal?

Resources

• Google SRE Workbook, Chapter 16, "Canarying Releases" — sre.google/workbook/canarying-releases. Describes the pattern of a canary with multiple stages, each with a bigger population than the previous one — the formal foundation for this lesson's four-stage ramp.
• Martin Fowler (bliki), "CanaryRelease" — martinfowler.com/bliki/CanaryRelease.html. The reference definition of canary release as a technique to reduce risk, exposing a small subset first before exposing the entire infrastructure.