Module 1: From Experiment To Launch

The launch as a risk decision, not a switch

Next →

Description

Lesson 2 made it clear that "the experiment won" doesn't equal "it's ready for 100%." This lesson answers the question that's left open: if launching isn't a switch you flip because the experiment did well, what is it? The answer you're going to build today is the backbone of this entire guide: launching is a risk decision, and like any risk decision, it's judged with two questions — if it goes wrong, how bad is it, and how easy is it to go back.

Connection to the module. This lesson takes the "why" from lesson 2 — the exposure gap — and gives it structure: a two-axis framework you'll use for the rest of the guide to decide how to launch anything, not just recommendations. Lesson 4 will put an exact number on the first axis (how bad, measured in affected users); lessons 5 and 6 will show which tools reduce the second axis (how easy it is to go back). Today you install the whole framework, before the rest of the module develops it piece by piece.

An analogy: the one-way door and the two-way door

Jeff Bezos, in a letter to Amazon shareholders, described two types of decisions. Some are one-way doors: once you walk through, it's hard or impossible to go back to how things were — think of signing a ten-year contract, or deleting a production database with no backup. Those decisions deserve slow deliberation, with lots of people consulted, because the cost of getting it wrong is high and permanent. Others are two-way doors: if you walk through and don't like what you see, you simply go back — think of trying a new meeting time for two weeks. Those decisions don't need the same heavy process; in fact, treating them as if they were one-way only makes the organization slow and averse to experimenting.

A product launch is almost never a pure door of a single type — and that's exactly the point. recommendations isn't a full one-way door (with the right tools, it can be turned off), but it isn't a frictionless two-way door either: while it's on, every second that passes, real people are having a slower experience than they should. The question you're going to learn to ask yourself isn't "is it reversible or not?" in black and white, but "how reversible is it, and how bad is it while you haven't reversed it" — the two coordinates that place any launch on a risk map.

Worked example: launchRiskProfile() on three Mercado launches

Let's build the simplest possible classifier with the analogy's two coordinates: how bad it is if it goes wrong (severityIfWrong) and how easy it is to go back (reversible). We'll run it on three real or hypothetical Mercado launches, including the one this guide follows:

// launchRiskProfile: classifies a launch by severity if something goes wrong x
// whether it's easy or hard to go back -- the same criterion that separates a
// "type 2" decision (two-way door, reversible) from a "type 1" one (one-way
// door, hard to undo).
function launchRiskProfile({ name, severityIfWrong, reversible }) {
  let strategy;
  if (reversible && severityIfWrong === 'low') strategy = 'ship directly, no ceremony';
  else if (reversible && severityIfWrong !== 'low') strategy = 'expose gradually, with rollback ready';
  else strategy = 'maximum caution: manual review before exposing anyone';
  return { name, severityIfWrong, reversible, strategy };
}

const launches = [
  { name: 'fix a typo in a product description', severityIfWrong: 'low', reversible: true },
  { name: 'recommendations (latency guardrail already broken)', severityIfWrong: 'high', reversible: true },
  { name: 'migrate the orders schema by dropping an old column', severityIfWrong: 'high', reversible: false },
];

console.log('=== launchRiskProfile on three Mercado launches ===\n');
launches.forEach((l) => {
  const r = launchRiskProfile(l);
  console.log(r.name);
  console.log('  severity=' + r.severityIfWrong + '  reversible=' + r.reversible + '  -> ' + r.strategy + '\n');
});

What to expect. Running the file with Node, the output is exactly this:

=== launchRiskProfile on three Mercado launches ===

fix a typo in a product description
  severity=low  reversible=true  -> ship directly, no ceremony

recommendations (latency guardrail already broken)
  severity=high  reversible=true  -> expose gradually, with rollback ready

migrate the orders schema by dropping an old column
  severity=high  reversible=false  -> maximum caution: manual review before exposing anyone

Notice where recommendations lands: not at the easy extreme (the typo, which ships with no ceremony because even if something goes wrong, it's trivial and reversible), nor at the most cautious extreme (the irreversible migration, which demands manual review before touching production). It lands in the middle zone — bad if it goes wrong, but reversible with the right tools — and that middle zone is precisely where almost every real product launch lives. It's not "ship without care" nor "never launch": it's "expose gradually, and keep the rollback ready while you do it." That sentence, taken seriously, is the whole argument of modules 2 through 5 of this guide.

Why "reversible" isn't the same as "costless"

A clarification is worth making before moving on, because it's a common misunderstanding: recommendations being "reversible" doesn't mean shipping it badly is free. Every minute that 250,000 users are exposed to checkout latency above the guardrail is a minute of a genuinely worse experience, with a real cost — abandonment, frustration, maybe support complaints — even if you eventually turn the feature off and everything "goes back to normal." Reversibility doesn't erase the damage that already happened while the feature was on; it only stops that damage from continuing to accumulate after you detect it. That's why the second axis of the framework — how bad it is if it goes wrong — matters as much as the first: the more people exposed before you decide to revert, the more already-occurred damage you'll have to explain, even if the rollback itself is instant. Lesson 4 puts an exact number on that relationship between "how many people you expose" and "how much damage already happened before you revert."

Common mistakes

Treating every launch with the same level of ceremony, regardless of its risk profile. What happens: a team demands the same heavy process of approvals, reviews, and meetings to launch a trivial text change as to launch recommendations with its broken guardrail. Why it happens: it's simpler to have one policy for "all launches" than to evaluate case by case, and nobody wants to be the one who decides what deserves less care. How to spot it: you ask "is this a one-way or two-way door, and how bad is it while we haven't reverted it?" and the answer is "I don't know, we apply the same checklist to everything." How to fix it: use launchRiskProfile()'s two-axis criterion to calibrate the level of care — a typo doesn't need the same process as an irreversible migration, and treating them the same only slows down the trivial without making the serious any safer.

Confusing "reversible" with "no consequences while it's active." What happens: someone argues "it doesn't matter if the guardrail breaks, we can turn it off whenever we want," as if the possibility of reverting eliminated the cost of every minute the regression was affecting real users. Why it happens: "reversible" sounds reassuring, and it's easy to jump from there to "so there's no real urgency." How to spot it: the conversation about reversibility never mentions how many people would be exposed, or for how long, before the rollback kicks in. How to fix it: as in the previous section, remember that reversibility stops future damage, it doesn't erase damage already done — the smaller the exposure before reverting, the smaller the accumulated damage you'll have to explain.

Using the risk framework to justify never launching. What happens: someone uses the existence of any severity — however low — as an excuse to indefinitely postpone a launch, without distinguishing between "severe and irreversible" and "severe but containable." Why it happens: it's more comfortable to avoid risk entirely than to learn to manage it carefully, especially if "managing the risk" doesn't yet have a concrete name (that arrives in modules 2 through 5). How to spot it: the conclusion of any risk analysis is always "better not to launch yet," regardless of the change's real profile. How to fix it: the launchRiskProfile() framework has three outputs, not two — "ship directly," "expose gradually with rollback ready," and "maximum caution." Most product launches, recommendations included, fall in the second category: neither excessive ceremony, nor a careless green light.

Exercises

Exercise 1 — Classify a new launch. Mercado's team wants to launch a color change on the "Add to cart" button (from blue to green), based on a previous experiment that showed a small lift in clicks. If something goes wrong, it's trivial to notice and to revert with a simple configuration change. What severityIfWrong and reversible would you assign, and what strategy would launchRiskProfile() return?

See solution

severityIfWrong: 'low' (a button color, in the worst case, marginally affects clicks — there's no risk to data, money, or a broken experience) and reversible: true (reverting a color is instant). With those two values, launchRiskProfile() returns 'ship directly, no ceremony' — the same result as the worked example's typo. This is exactly the kind of launch where demanding the same heavy process as recommendations would be this lesson's first common mistake: unnecessary ceremony for a low-risk change.

Exercise 2 — The middle case, with nuance. Suppose Mercado has, today, zero feature flag infrastructure — there's no way to turn recommendations off without a full new deploy (which takes 40 minutes). Is it still reversible: true in the sense launchRiskProfile() uses? Explain your answer.

See solution

It's an intentionally uncomfortable gray zone, and that discomfort is the point. Technically, reverting a full deploy is possible — it's not an absolute one-way door, the way an irreversible migration that deletes data with no backup would be — but 40 minutes to revert is an eternity if 250,000 users are exposed to a latency regression in the meantime. The honest answer: it's "reversible, but slowly" — a third zone between the worked example's two extremes, and precisely the reason module 2 of this guide teaches feature flags as a specific mechanism to make the rollback nearly instant, instead of depending on a new deploy. Without that tool, even a launch that's "reversible in theory" behaves, in practice, much more like one with sustained high severity.

Exercise 3 — Apply the framework to your own context. Think of a real change you've launched (or seen launched) at your job — it can be code, a policy, a process. Using this lesson's two questions (how bad if it goes wrong?, how easy to go back?), which zone of the framework did it land in? Did the level of care it launched with match that zone, or was there a mismatch (too much ceremony for something trivial, or too little care for something serious)?

See solution

There's no single answer — it depends on your experience — but here's the quality check: if your change was severe and hard to revert, and it launched with the same level of care as a trivial change, that's a real mismatch worth naming in retrospect. And if it was trivial and reversible, but went through weeks of approvals, that's the reverse mismatch — the kind that makes a team lose speed without gaining real safety. The point of this exercise isn't to judge the past decision, but to practice the habit of explicitly naming the two coordinates before deciding how much care a launch deserves, instead of deciding it by habit or instinct.

Summary and next step

In this lesson you reframed the launch as a risk decision with two coordinates — how bad if it goes wrong, how easy to go back — instead of a binary switch. With launchRiskProfile() you placed three different launches on that map, and saw where recommendations lands: severe if it goes wrong (the latency guardrail already broken), but reversible with the right tools — the zone where the correct strategy is neither "ship without care" nor "never launch," but "expose gradually, with the rollback ready."

Before moving on you should be able to: explain the risk framework's two coordinates in your own words; place a new launch in the framework's three zones; and distinguish between "reversible" and "no consequences while it's active."

Lesson 4 takes the severity coordinate and turns it into an exact number: how many real users end up exposed to a known regression, depending on what fraction of the base receives the launch first. It's the concept that will accompany you for the rest of this guide — the blast radius, run in Node on the full recommendations case.

Resources

← PreviousWinning the experiment is not the same as shipping the productNext →Blast radius: how many real people a known error touches