Module introduction: watching the ramp, not just climbing it

Why this module, now

Module 3 gave you the full ramp: rolloutPlan() defines the four stages — canary 1% → 10% → 50% → 100% — and the explicit criteria for advancing from one to the next. With that solved, it looks like everything's already there: you have the switch (module 2) and you have the plan for how many people to expose at each step (module 3). But module 3 deliberately left a question open, and left it open with these exact words at its boundary: "watching guardrails during the ramp is M4." Designing the staircase and climbing the staircase are two different problems. This module solves the second one: while you climb, what do you watch, and when do you stop climbing?

The answer isn't entirely new. The metrics guide already built guardrailCheck() and already defined Mercado's four business guardrails: checkoutLatencyP95Ms (ceiling 800ms), complaintRate, churnRate, and grossMargin. That work doesn't get repeated here — it gets reused, unchanged, and put to work live, stage by stage, as the real rollout advances. This module's new piece is guardrailWatch(): the function that walks module 3's ramp and decides, at each stage, whether the guardrail holds up (continue) or whether it's time to stop right there (HALT) — before the next stage exposes even more people to a problem already known to exist.

Connection to module 3. That module's lesson 8 ended with the ramp designed and its advance criteria defined, but without yet running any real guardrail data against those criteria — the rollout, at that point, hadn't started yet. This module takes that same ramp and runs it against recommendations' launch's real numbers: the latency the metrics guide had already measured at 910ms, well above the 800ms ceiling.

An analogy: the car's dashboard, not just the speedometer

Accelerating a car on a long highway feels, most of the time, like a single number: speed. But no experienced driver looks only at the speedometer. The dashboard has, right next to it, engine temperature and oil pressure — two numbers that almost never change anything on a normal trip, and that exist exactly for the day they do. An overheating engine doesn't warn you first with total failure: it warns you with the temperature needle climbing, minutes before something actually melts. The driver who only watches the speedometer — "I'm going fast, we're doing fine" — and never the temperature, is the one who ends up stranded on the shoulder, engine ruined, wondering what happened.

recommendations' rollout has the same pair of needles. The speedometer is checkoutConversionRate, the primary metric the metrics guide confirmed improves, strongly, as the rollout advances. The engine temperature is checkoutLatencyP95Ms — and in this concrete case, we already know it's rising more than it should. This module is about learning to read that second needle with the same discipline as the first, and to brake the car — not turn it off, brake it — before the needle reaches the red zone, even while the speedometer keeps showing a number that looks spectacular.

The map of this module's eight lessons

Lesson    Question it answers
────────  ──────────────────────────────────────────────────────────────
L1        (this one) Where did module 3 leave us, and what's left to watch?
L2        What do you watch as a ramp climbs: signals and their speed?
L3        How do you watch a guardrail LIVE, stage by stage?
L4        When do you stop, by what criterion, and what happens if you don't?
L5        What should an in-progress launch's dashboard show?
L6        How do you design an alert that's actually useful, not one that just makes noise?
L7        How much regression room do you have left before you must stop?
L8        Project: watch recommendations' rollout and decide to stop

Lesson 2 separates two types of signal that tend to get confused: those that already tell you something useful with few exposed users (checkoutLatencyP95Ms, measurable on every request) and those that need much more volume or time to say something trustworthy (churnRate, complaintRate). Lesson 3 builds the module's central piece, guardrailWatch(), and runs it on recommendations' real rollout: at canary, latency holds up; climbing to 10%, it breaks. Lesson 4 stays with that result and asks what "stopping" means in practice — and what would happen to Mercado's user base if the team decided to ignore the HALT and keep climbing anyway. Lesson 5 designs the dashboard that makes visible, at a glance, the state guardrailWatch() calculates. Lesson 6 gets into alerts: the difference between one that flags a loose technical symptom and one that ties that symptom to real user impact. Lesson 7 closes the topical arc with the error budget: how much latency regression room you have left, in concrete milliseconds, before the budget runs out. Lesson 8, the project, reuses guardrailWatch() without changing a line on the full case: canary at 10%, and the decision to stop before reaching 50%.

The boundary: what this module does NOT cover

This module watches. It doesn't build the ramp, doesn't decide what to do after stopping, and doesn't calculate whether the primary result is statistically real:

• How the ramp is designed — the four stages, their advance criteria — is module 3. Here the ramp already exists; this module runs it against live guardrail data.
• What to do after a HALT — revert, fix forward, the runbook, incident severity — is module 5. This module stops exactly at the moment of the decision to stop; the full process of responding to the problem starts at the next door.
• The postmortem for what happened is module 6. Here you observe and decide to stop; documenting the root cause and contributing factors is later work.
• Whether the lift in checkoutConversionRate is statistically significant was already answered by the metrics guide (p=0.0114, significant). This module doesn't recalculate that statistic — it takes it as known, and focuses on the guardrails watched while the primary number climbs.

Common mistakes

Believing "watching" means looking at the dashboard once something has already gone wrong. What happens: the team designs the ramp, turns on the rollout, and only opens the guardrail dashboard when someone else — a user, a support ticket — reports something feels slow. Why it happens: between building the ramp (module 3) and watching it live there's a different discipline, and it's easy to assume "it's all automated already" without having decided who watches what, and when. How to spot it: if nobody can say, for the rollout's current stage, what checkoutLatencyP95Ms's most recent value is, watching exists on paper, not in practice. How to fix it: as this module builds it, watching is a function that runs at every stage, not an occasional check — lesson 3's guardrailWatch() is, literally, that discipline turned into code.

Thinking a single broken guardrail invalidates the entire launch. What happens: someone sees HALT at the 10% stage and concludes recommendations has to be canceled entirely, without distinguishing between "stop the ramp here" and "revert everything already launched." Why it happens: the word HALT sounds final, and it's simpler to react in black and white than to hold the distinction between pausing and canceling. How to spot it: the conversation jumps from "the guardrail broke" to "we have to revert now" skipping "what stage are we at, and how bad is it to stay there while we diagnose?" How to fix it: this module treats HALT as "stop before exposing more people" — not as "revert what's already exposed." The decision to revert, with its whole process, belongs to module 5.

Confusing "the guardrail hasn't broken yet" with "the guardrail is confirmed healthy." What happens: at the canary stage, with very few exposed users, every guardrail comes back OK, and the team reads it as strong confirmation that everything's fine. Why it happens: OK looks equally definitive no matter how many users back that number up, and it's easy to miss the difference between "no sign of a problem" and "confirmed signal that there's no problem." How to spot it: if nobody can say how many users or events back up an OK at an early stage, that OK might simply mean there still isn't enough volume for the guardrail to move. How to fix it: this module's lesson 2 builds exactly this distinction — which guardrails already have useful signal at canary, and which don't yet.

Exercises

Exercise 1 — Locate the question. For each question, say whether module 3 (already covered), this module 4, or a later module (5 or 6) answers it:

• (a) "What percentage should recommendations' rollout start at, and what percentage does it climb to next?"
• (b) "At 10% rollout, latency broke its ceiling. Do we keep going to 50% or stop here?"
• (c) "We already decided to stop at 10%. Do we revert the 10% already exposed, or leave it while we fix the root cause?"

See solution

• (a) Module 3 — rolloutPlan() designed exactly those stages and their advance criteria.
• (b) This module 4 — lesson 3's guardrailWatch() answers exactly this question, running the guardrail against each stage.
• (c) Module 5 — deciding between reverting and fixing forward, with its runbook, is rollback and incident response's territory.

Exercise 2 — The analogy, in your words. Using the car dashboard analogy, explain in two sentences why a driver who only watches the speedometer can end up with a ruined engine, even while going "fine" by that single number.

See solution

The speedometer only tells you how fast you're going, not whether something in the system is degrading while you go fast — engine temperature can rise silently, with speed not reflecting it at all. By the time the engine fails, it's already too late to have done anything with just that one needle: the useful moment to act was earlier, when temperature started rising but the engine still worked. It's exactly the same problem as a team that only watches checkoutConversionRate while checkoutLatencyP95Ms degrades underneath, with the primary number not showing it at all.

Exercise 3 — Prediction. Without having read lesson 3 yet, what do you think should happen if, on reaching the 50% stage, the latency guardrail had already broken at the previous stage (10%)? Should guardrailWatch() keep evaluating the 50% stage with real data, or do something else?

See solution

If the guardrail already broke at a previous stage, there shouldn't be any real data from the 50% stage to evaluate — because the team, following this module's discipline, should never have advanced the rollout that far. guardrailWatch() is going to mark that stage as "not reached," instead of pretending it evaluated something that, in practice, never happened. It's the same logic you see in the analogy: if the temperature already rose too much at kilometer 10, you don't keep accelerating to kilometer 50 "to see what happens."

Summary and next step

In this lesson you located this module within the guide's full arc: module 3 designed the ramp (canary 1% → 10% → 50% → 100%); this module 4 watches it, stage by stage, with the same business guardrails the metrics guide already defined — latency, complaints, churn, margins. You saw the car dashboard analogy — the speedometer versus engine temperature — and the map of the eight lessons ahead, with its clear boundary against module 5 (what to do after stopping) and module 6 (the postmortem).

Before moving on you should be able to: explain why designing the ramp and watching it are two different problems; distinguish "stopping the ramp" from "reverting what's already exposed"; and anticipate why an OK guardrail at a low-user stage isn't, yet, strong confirmation.

Lesson 2 starts at the beginning: before watching anything, you need to know what signals exist and how fast each one starts saying something useful — the difference between a signal that warns in minutes and one that takes weeks to accumulate enough evidence.

Resources

• Google SRE Book, Chapter 6, "Monitoring Distributed Systems" — sre.google/sre-book/monitoring-distributed-systems. The chapter that formalizes the four golden signals — latency, traffic, errors, saturation — this module applies, in business form, to recommendations' rollout.
• LaunchDarkly, "Introducing Guardrail Metrics: best-practice metrics for every release" — launchdarkly.com/blog/introducing-guardrail-metrics. How a real feature flag platform automates exactly the watching this module builds by hand: guardrails tied to every release, reviewed at every stage.
• Google SRE Book, Chapter 3, "Embracing Risk" — sre.google/sre-book/embracing-risk. The chapter on error budgets, this module's lesson 7's central idea: how much regression you can afford before the budget runs out.