Iterating on the winner: fix, relaunch, measure again

Description

The previous lesson left the second cycle's ship described in a single sentence: "call redesigned to async mode + caching; relaunched through the same ramp." This lesson executes that ship in detail. With the postmortem's three action items already resolved — the call to the recommendation engine no longer blocks, caching exists for the most-requested recommendations, the ramp's advance criteria already requires a load test — recommendations goes through exactly the same ramp module 3 designed: canary 1% → 10% → 50% → 100%. This lesson's question is simple to state and satisfying to answer: did the fix work?

Connection to the module. This lesson reuses rolloutPlan() without changing a line from module 3 — the same pattern you've already seen with guardrailWatch(), isEnabled(), and buildPostmortem() throughout this guide: a function gets built once, and gets reused every time the case calls for it, without rewriting it. The only thing that changes between the original rollout (module 3, lesson 8) and this relaunch is the measured data at each stage — the decision logic is identical.

An analogy: the second test drive, after the adjustment

When a car manufacturer detects a problem in track testing — say, the brakes overheat more than expected on repeated tight corners — it doesn't scrap the entire model or start the design from scratch. It adjusts the brake system — a different compound, a bigger disc, better ventilation — and reruns exactly the same battery of tests, in the same order, under the same conditions. It doesn't invent a new track for the second test, or skip steps because it "already learned the lesson" — it runs the same test ramp, now with the fix in place, precisely because that ramp is what revealed the problem the first time, and it's the best tool available to confirm the fix works under the same conditions that exposed it.

recommendations's relaunch is exactly that second test drive. The ramp doesn't change — it's still canary 1% → 10% → 50% → 100%, with the same 800ms latency ceiling —; what changes is that, this time, the system going through it already has the fix in place.

Worked example: rolloutPlan() on recommendations's relaunch

// rolloutPlan: UNCHANGED from module 3. Walks the stages IN ORDER and
// decides ADVANCE/HOLD based on each one's criteria.
function rolloutPlan(stages) {
  const results = [];
  let halted = false;
  for (const s of stages) {
    if (halted) { results.push({ ...s, decision: 'NOT_REACHED' }); continue; }
    const decision = s.advanceIf(s.measured) ? 'ADVANCE' : 'HOLD';
    results.push({ ...s, decision });
    if (decision === 'HOLD') halted = true;
  }
  return results;
}

const ceiling = 800; // the same ceiling as always; it didn't change with the fix

// The SAME ramp from module 3, run again after the postmortem fix
// (async call + caching for the recommendation engine).
const relaunchPlan = [
  { percent: 0.01, label: 'canary 1%', measured: { p95Latency: 674 }, advanceIf: (m) => m.p95Latency <= ceiling },
  { percent: 0.10, label: 'rollout 10%', measured: { p95Latency: 738 }, advanceIf: (m) => m.p95Latency <= ceiling },
  { percent: 0.50, label: 'rollout 50%', measured: { p95Latency: 762 }, advanceIf: (m) => m.p95Latency <= ceiling },
  { percent: 1.00, label: 'rollout 100%', measured: { p95Latency: 781 }, advanceIf: (m) => m.p95Latency <= ceiling },
];

console.log('=== recommendations relaunch, after the fix (p95 ceiling=' + ceiling + 'ms) ===\n');
const results = rolloutPlan(relaunchPlan);
results.forEach((s) => {
  console.log(s.label.padEnd(14) + 'p95=' + String(s.measured.p95Latency).padStart(4) + 'ms  -> ' + s.decision);
});

const allAdvanced = results.every((s) => s.decision === 'ADVANCE');
console.log('\nThe full ramp advanced with no HOLD at all: ' + allAdvanced);

What to expect. Running the file with Node, the output is exactly this:

=== recommendations relaunch, after the fix (p95 ceiling=800ms) ===

canary 1%     p95= 674ms  -> ADVANCE
rollout 10%   p95= 738ms  -> ADVANCE
rollout 50%   p95= 762ms  -> ADVANCE
rollout 100%  p95= 781ms  -> ADVANCE

The full ramp advanced with no HOLD at all: true

Compare this result with module 3's: there, the same ramp stopped at rollout 10% with p95Latency=910ms, well above the ceiling. This time, the 10% stage measures 738ms — under the ceiling, and in fact lower than the number that had shown up in canary the first time (910ms at the 10% stage of the original attempt, against 738ms now). The fix didn't just avoid repeating the problem — it left real margin: even at rollout 100%, with the whole base exposed, p95Latency reaches 781ms, still 19ms below the ceiling. The full ramp advances, stage by stage, without a single HOLD — the first concrete confirmation that the postmortem's three action items (lesson 3) attacked the right cause.

Going deeper: why reuse the SAME ramp, instead of designing a "more careful" new one

It might seem, at first glance, that after an incident it's better to design a slower ramp for the relaunch — smaller stages, more wait time between each one — as a way of "being more careful" the second time. It's worth resisting that impulse, and the reason is the same one that explains why the car manufacturer in the analogy doesn't invent a new track: the original ramp already proved it works for detecting the exact problem that showed up — it stopped at 10% the first time, exactly where it should have. Designing a different ramp for the relaunch doesn't make it safer; it makes it different, which means you no longer know, with the same certainty, how it would have behaved against the original problem.

This doesn't mean the ramp should never change — in fact, one of the postmortem's three action items (lesson 3) was, precisely, adding a load test to the advance criteria, a real change to the ramp. The difference is that this change is justified by a specific contributing factor, documented in the postmortem, not by a general feeling of "let's go slower just in case." Changing the ramp without a reason traceable to a contributing factor is, at bottom, the same kind of evidence-free decision module 1 already warned against making with guardrail thresholds: it looks prudent, but it isn't anchored to anything concrete.

Common mistakes

Relaunching without having confirmed the postmortem's three action items are already closed. What happens: the team, eager to recover recommendations's business value, relaunches the ramp before the call's redesign, the caching, or the load test in the advance criteria are fully implemented. Why it happens: business pressure for a result already known to win (+18.75% on conversion) is real, and waiting for all three action items to close feels like a costly delay. How to spot it: if asking "is the recommendations cache already deployed?" gets the answer "not yet, but it should be fine anyway," the relaunch is happening with the root cause only partially resolved. How to fix it: the postmortem's three action items exist because the three contributing factors together caused the incident — relaunching with only one or two resolved is betting that the remaining factors won't combine the same way again, with no real evidence that's true.

Interpreting the ramp advancing cleanly as "no need to keep monitoring." What happens: seeing allAdvanced: true, the team disables module 4's monitoring dashboard, assuming a successful relaunch means the risk is completely gone. Why it happens: four stages in a row at ADVANCE feel like a definitive confirmation, and it's tempting to treat "passed the ramp" as "nothing left to ever watch again." How to spot it: if nobody can say what recommendations's p95Latency was last week — only the relaunch-day figure — monitoring was abandoned too soon. How to fix it: module 4's guardrailWatch() wasn't a one-time tool for the rollout — it remains the correct discipline for any feature in production, indefinitely, though with lower urgency once the known risk has been contained.

Confusing "latency held" with "the winner is fully confirmed." What happens: the team celebrates the successful relaunch and closes recommendations's case entirely, without connecting this result to the question lesson 5 explicitly left open: does the conversion lift hold, or was it a novelty effect? Why it happens: latency was the visible, dramatic problem — the one that caused the incident — so solving it feels like solving "the problem" in general, leaving aside that the guardrail and the primary metric are different questions. How to spot it: if this relaunch's report doesn't mention any plan to remeasure checkoutConversion in the following weeks, the ship → measure → learn loop's cycle stayed incomplete at the measure part. How to fix it: this lesson resolves the guardrail (p95Latency), not the primary metric (checkoutConversion) — lesson 7 builds exactly the missing check to close that second question.

Exercises

Exercise 1 — Compare the two rollouts, number by number. Using module 3's data (canary 720ms, rollout 10% 910ms) and this lesson's (canary 674ms, rollout 10% 738ms), calculate how much p95Latency dropped at the 10% stage, in milliseconds and as a percentage.

See solution

The drop in milliseconds: 910 - 738 = 172ms. As a percentage: 172 / 910 * 100 ≈ 18.9% reduction from the original value that broke the guardrail. It's worth noting something interesting: the canary value also dropped (from 720ms to 674ms, a 46ms reduction, ≈6.4%), even though the guardrail had never broken at canary — the fix (async call + caching) improved latency at every stage, not just the one that had failed, because it attacked a cause that affects any traffic volume, but only became critical past a certain volume.

Exercise 2 — Simulate a relaunch that isn't ready yet. Suppose only the caching (one of the three action items) got implemented, but the call to the recommendation engine is still blocking. With that partial data, latency at rollout 10% improves, but only drops to 830ms (still above the 800ms ceiling). What decision would rolloutPlan() give at that stage, and which of this lesson's "Common mistakes" does this scenario illustrate?

See solution

With p95Latency: 830 and ceiling: 800, the condition 830 <= 800 is false, so rolloutPlan() would flag that stage as HOLD, stopping the ramp exactly like the original attempt — though with a lower value than before (830ms versus 910ms), it still breaks the guardrail. This scenario illustrates the first common mistake: relaunching with only part of the action items resolved can improve the number without fully solving the problem, and the guardrail — which doesn't negotiate with "it improved, though not enough" — catches it just like the first time.

Exercise 3 — Defend the decision to reuse the same ramp. A teammate proposes designing a completely new ramp for the relaunch, with smaller stages (0.5% → 5% → 25% → 100%) "to be more conservative after what happened." In 2-3 sentences, explain why reusing the original ramp, with the specific adjustment to the advance criteria (the load test), is preferable to designing a new ramp from scratch.

See solution

A reasonable answer: "The original ramp (canary 1% → 10% → 50% → 100%) already proved it works for detecting exactly the kind of problem we had — it stopped at the right stage the first time. Designing a new ramp doesn't make it safer, it makes it different, and we lose the certainty that it behaves just as well against the same kind of regression. The only change that is justified is the one we already identified in the postmortem: adding the load test to the advance criteria — a specific adjustment with a documented cause, not a full redesign based on a general feeling of caution."

Summary and next step

In this lesson you executed the concrete iteration on the winner: reusing rolloutPlan() unchanged, the relaunched recommendations — with the call to the recommendation engine already async, with caching, and with the advance criteria already fixed — went through the same ramp's four stages from module 3 with not a single HOLD, with p95Latency holding between 674ms and 781ms across every stage, well below the 800ms ceiling. You confirmed the postmortem's three action items attacked the right cause, and saw why reusing the original ramp — with its one documented adjustment — is preferable to designing a new one "just in case."

Before moving on you should be able to: explain why reusing the same ramp for the relaunch has more value than designing a new one; distinguish "the guardrail held" from "the winner is fully confirmed"; and anticipate why the question about checkoutConversion is still unanswered, even though p95Latency is already resolved.

Lesson 7 closes that pending question: now that latency stopped being a risk, does the conversion lift hold week after week, or was it, in part, a novelty effect — users' initial enthusiasm for something new, which fades over time?

Resources

• Google SRE Workbook, Chapter 16, "Canarying Releases" — sre.google/workbook/canarying-releases. The same reference from module 3, relevant here because it confirms why running the same canary/rollout battery after a fix, instead of a new one, is the recommended practice for validating a correction with the same rigor that detected the problem. In English.
• Charity Majors, "Deploys Are the WRONG Way to Change User Experience" — honeycomb.io/blog/deploys-wrong-way-change-user-experience. The same article cited in module 1, relevant here because its central argument — decoupling deploy from release — is what lets you relaunch an already-fixed feature without a new infrastructure deployment, just by changing the flag's state. In English.
• LaunchDarkly, "What Is Progressive Delivery All About?" — launchdarkly.com/blog/what-is-progressive-delivery-all-about. Industry context on why the same gradual ramps that protect a first launch are the right tool for validating a later iteration on the same change. In English.