Rollback is your safety net

Description

recommendations already has a kill switch from module 2: a single line, recommendationsFlag.enabled = false, that shuts off exposure for 100% of users instantly, no matter what stage of the rollout the ramp was at. This lesson gives that capability a broader name, and puts it in the context where it truly matters: when a guardrail breaks in production, the kill switch is your rollback — and it's almost always the fastest option available.

Connection to the module. Lesson 1 left the question "do we revert or fix forward?" open, without giving any criteria yet to decide it. This lesson doesn't answer that full question — that's lesson 3 — but it builds half of the information needed to answer it: how much reverting costs, in time. Without that number, there's no way to compare it against the cost of a forward fix.

An analogy: the trapeze artist's net, not Ctrl+Z

It's tempting to think of rollback as the Ctrl+Z of a text editor: undo the last change and go back exactly to where you were. That analogy isn't entirely wrong, but it's missing an important part — a Ctrl+Z undoes one change, typically the most recent, and assumes everything else stayed intact in the meantime. The rollback of a real incident is more like the trapeze artist's net: it doesn't stop the trapeze artist from slipping off the trapeze — the guardrail already broke, that already happened —; what it does is stop that slip from turning into a fall to the floor that hurts someone. The net doesn't rewind time to before the slip. It catches the fall, at the moment it happens, and buys time to think through the next move without anyone getting hurt in the meantime.

recommendationsFlag.enabled = false is exactly that net. It doesn't erase the fact that latency broke during the minutes or hours the feature was exposed to 10% of the base — that damage, on those 25,000 users, already happened. What it does is immediately stop it from happening to one more user. The net doesn't prevent the fall. It prevents the fall from turning into something worse.

Worked example: how long each way of "shrinking" exposure takes

Mercado has, in theory, two ways to reduce recommendations exposure once the guardrail is broken at rollout 10%: flip off the kill switch in one shot, or "walk back" the ramp one step at a time, with the same care used to climb it. Let's measure the real cost of each:

// revertLatency: compares two ways of "shrinking" recommendations exposure
// when the guardrail breaks at rollout 10% (M3/M4). killSwitch uses the same
// `enabled` field from M2 L5 -- a boolean, regardless of stage. stepDown would try
// to walk back one ramp step at a time, respecting the same minimum dwell time
// (M3 L7) used to CLIMB -- a protocol designed to advance with confidence, not
// for an emergency.
function revertLatency(mechanism) {
  const dwellMinutesAtCurrentStage = 12 * 60; // minDwellHours for "rollout 10%" (M3 project)
  if (mechanism === 'killSwitch') return 2; // a boolean, enabled=false, regardless of stage
  if (mechanism === 'stepDown') return dwellMinutesAtCurrentStage; // wait out the dwell time before trusting the step back
  throw new Error('unknown mechanism: ' + mechanism);
}

console.log('=== revertLatency at rollout 10% (25,000 users exposed, guardrail broken) ===\n');
['killSwitch', 'stepDown'].forEach((m) => {
  const minutes = revertLatency(m);
  const label = minutes >= 60 ? (minutes / 60) + 'h' : minutes + 'min';
  console.log(m.padEnd(12) + '-> ' + label + ' until no user sees recommendations');
});

What to expect. Running the file with Node, the output is exactly this:

=== revertLatency at rollout 10% (25,000 users exposed, guardrail broken) ===

killSwitch  -> 2min until no user sees recommendations
stepDown    -> 12h until no user sees recommendations

The difference is enormous, and it's worth understanding where it comes from. killSwitch takes 2 minutes because, as you saw in module 2, it's literally flipping a value from true to false — it doesn't matter whether the ramp was at 1% or 100%, the cost is the same. stepDown, on the other hand, takes 12 hours, because that number isn't arbitrary: it's the same minDwellHours module 3 assigned to the rollout 10% stage — the minimum time you need to wait before trusting that a stage's data is real, not noise. That wait time makes complete sense when you're advancing carefully toward more exposure. It makes no sense at all when you're trying to exit an exposure you already know is harmful — and yet, if your only mechanism for reducing the percentage is "step it down, wait, confirm, step it down again," you end up trapped in the same slow protocol designed for the opposite of what you need at that moment.

Why the kill switch is the net, and not an occasional plan B

Notice something important about the result: killSwitch doesn't depend on what stage the ramp is at. If the guardrail had broken at rollout 50% instead of rollout 10%, the cost of reverting with the kill switch would still be 2 minutes — while stepDown would have taken 24 hours (the minDwellHours for that stage, per module 3's project table), not 12. That's the property that makes the kill switch a real safety net and not just a useful tool: its cost doesn't grow with the blast radius the problem has already reached. The higher the ramp climbs, the more you need an exit that doesn't depend on the same caution used to climb it.

This doesn't mean rollback is always the right answer — that's exactly what lesson 3 is going to put to the test, comparing its cost against a forward fix's —. It means that, before comparing anything, you need to know the real number: in this guide, thanks to module 2's feature flag, that number is almost always minutes, not hours.

Common mistakes

Confusing "lowering the percentage" with "activating the safety net." What happens: facing a broken guardrail, someone reduces rolloutPercent from 10 to 5, thinking that counts as "reverting a bit" — when in reality, depending on how isEnabled() (module 2) is implemented, that change still exposes half of the same users to the same broken latency. Why it happens: lowering a number feels like a rollback action, even though technically it doesn't shut anything off for anyone already exposed in a deterministic way. How to spot it: if after "reverting" there are still real users seeing the broken latency, the net wasn't activated — the dial was adjusted. How to fix it: as in this lesson's example, the real safety net is the kill switch — enabled = false — not an incremental adjustment of the percentage.

Measuring the kill switch's cost only at the stage you're already at, without noticing it's constant. What happens: someone calculates that the kill switch takes 2 minutes at rollout 10%, and assumes — without checking — that this number will grow if the problem is detected later, at a stage with more users exposed. Why it happens: it's intuitive to think "reverting something bigger" takes more time, as would happen with stepDown. How to spot it: the corrective question is "does this mechanism's cost depend on how many users are exposed, or not?" If the answer is "it doesn't depend," as happens with a boolean, the cost is constant regardless of stage. How to fix it: verify, as this lesson's example did, that the reversion mechanism doesn't scale with the size of exposure — that's precisely the property that makes it a reliable safety net.

Treating the kill switch's existence as an excuse to never calculate the cost of a forward fix. What happens: since the kill switch is so fast, a team starts assuming it's always the best option, without ever considering whether a forward fix might, in some particular case, be just as fast or more appropriate. Why it happens: 2 minutes is such a low number that comparing it against anything else feels like a waste of time. How to spot it: if no one on the team can name a situation — even a hypothetical one — where it would be worth not using the kill switch, the team is probably not thinking about the real cost of each option, just following habit. How to fix it: lesson 3 shows a real case, with data, where the cost of reverting outweighs a bounded fix's — the kill switch is the fast option by default, not the automatic option with no exceptions.

Exercises

Exercise 1 — Calculate the cost at the rollout 50% stage. Using the same logic as revertLatency(), and knowing that minDwellHours for rollout 50% is 24 hours (module 3's project table), how long would stepDown take if the guardrail had broken at that stage instead of rollout 10%? And killSwitch?

See solution

stepDown would take 24 hours — the dwellMinutesAtCurrentStage corresponding to rollout 50%, double what it was at rollout 10%. killSwitch would still take exactly 2 minutes, unchanged, because its cost doesn't depend on dwellMinutesAtCurrentStage at all — the function doesn't even look up that value when mechanism === 'killSwitch'. This is the lesson's central property: the higher the ramp climbs, the bigger the gap between the two options grows, and the clearer it becomes why the safety net has to be something that doesn't depend on the stage.

Exercise 2 — Find the weak point in the argument. Someone on the team objects: "the kill switch shuts off recommendations, but it doesn't fix the slow query that caused the broken latency — so technically we haven't 'reverted' anything, we just hid the symptom." Are they right? In what sense are they, and in what sense does that objection misunderstand what a rollback is designed to do?

See solution

They're right in a literal, limited sense: the kill switch doesn't touch the slow query's code, so the root cause still exists in the system after exposure is shut off. But the objection misunderstands the purpose of rollback: a rollback never promised to fix the root cause — it promised to stop the damage while the root cause is investigated calmly, without the pressure of users being exposed at that very moment. Confusing "reverting the visible damage" with "fixing the underlying problem" is exactly the mistake module 2 already named about the kill switch in general, and that this lesson repeats in the specific context of an incident: shutting off exposure buys time — it isn't, on its own, the complete solution.

Exercise 3 — Explain the net without using the word "rollback." In two or three sentences, explain to someone from another team why Mercado needs a way to "shrink" a feature's exposure that doesn't take longer the more people are already seeing it. You can use the trapeze artist analogy.

See solution

One example answer: "We need a way to shut off a feature that takes the same amount of time no matter how many people it's reaching at that moment — like the net under a trapeze artist, which catches the fall no matter what height it happened from. If our only way to shut something off depended on stepping it down gradually, with the same care we use to ramp it up, then the more successful the launch had been — the higher it had climbed before breaking — the longer it would take us to protect ourselves from a problem we already know about, right when we have the least time to spare." The central idea: the safety net has to be fast precisely at the worst moment — when the problem has already reached more people, not fewer.

Summary and next step

In this lesson you named rollback for what it is in the context of this guide: module 2's kill switch, now understood as a safety net — not a Ctrl+Z that rewinds time, but a mechanism that stops ongoing damage, regardless of what stage of the ramp the guardrail broke at. With revertLatency() you measured the real contrast: 2 minutes with the kill switch versus 12 hours if the only mechanism available were stepping the ramp back one stage at a time, with the same caution used to climb it.

Before moving on you should be able to: explain why the kill switch's cost doesn't depend on the rollout stage; distinguish "shutting off exposure" from "fixing the root cause," as two different things a rollback doesn't promise to conflate; and calculate, for any stage of the ramp, how long each reversion mechanism would take.

You already have half the information needed to decide between reverting and fixing forward: the cost of reverting. Lesson 3 builds the other half — the cost of a forward fix — and brings them together in rollbackDecision(), the model that makes the full decision on the real recommendations case.

Resources

• LaunchDarkly, "What Is a Kill Switch in Software Development?" — launchdarkly.com/blog/what-is-a-kill-switch-software-development. The same article from module 2, now in the context of a real incident: it explicitly describes the kill switch as the fastest reversion mechanism available when something goes wrong in production. In English.
• Charity Majors, "Deploys Are the WRONG Way to Change User Experience" — honeycomb.io/blog/deploys-wrong-way-change-user-experience. The central argument for why a flag reverts a user experience in minutes, while reverting at the code-deployment level can take much longer — the basis for this lesson's comparison. In English.
• Google SRE Workbook, Chapter 16, "Canarying Releases" — sre.google/workbook/canarying-releases. The same reference from module 3's project: the dwellMinutesAtCurrentStage this lesson reuses as stepDown's cost comes directly from the cautious-advance protocol this chapter describes. In English.