Project: write `recommendations`'s postmortem and plan its iteration

Description

This module's seven lessons built, piece by piece, everything needed to close the full learning cycle for an incident: how to investigate the system without blaming anyone (L2), how to structure that investigation into timeline, contributing factors, and action items (L3), why language decides whether that structure serves its purpose (L4), the ship → measure → learn loop framework that gives all of this meaning (L5), how to concretely iterate on the winner once the cause is fixed (L6), and how to confirm whether that winner holds up over time or was a novelty effect (L7). This mini-project puts you in the exact position of Mercado's team, weeks after the incident: writing the full postmortem for recommendations's latency regression, and planning — with data, not hope — its full iteration through to final confirmation.

Connection to the module. This project doesn't introduce any new concept: it reuses buildPostmortem() exactly as it stood at the end of lesson 3, rolloutPlan() exactly as it stood in module 3 and got reused in lesson 6, and noveltyCheck() exactly as it stood at the end of lesson 7 — all three, without changing a line. It's, literally, a rehearsal of the real work this entire module prepared you for: not memorizing that "postmortems must be blameless" in the abstract, but producing the full document and the iteration evidence on Mercado's concrete case.

An analogy: the full investigation report, from the black box to the next approved route

An aviation investigation report doesn't end at the black box analysis. After reconstructing the timeline and the systemic factors, the investigation body produces concrete safety recommendations — design changes, procedure changes, training changes —, and only once those recommendations are implemented does the aircraft (or the entire model) get recertified for passenger flights. Certification isn't a separate step from the report: it's the direct, verifiable consequence of the recommendations having been met and of the original problem no longer showing up in later tests.

This project follows exactly that same full chain: recommendations's postmortem (the reconstructed black box), the action items that come out of it (the safety recommendations), and the verification that, with those changes in place, both the original risk (latency) and the benefit that was being sought (the conversion lift) hold up in real production — the full "recertification" before declaring the case closed.

Part 1 — The complete blameless postmortem

Reusing buildPostmortem() unchanged, with the full latency incident:

// buildPostmortem: UNCHANGED from lesson 3.
function buildPostmortem(incident) {
  const timeline = [...incident.events].sort((a, b) => a.time.localeCompare(b.time));
  const namedPeople = ['Ana', 'Bruno', 'Carla', 'Diego', 'Elena'];
  const factors = incident.contributingFactors.map((description) => {
    const redFlag = namedPeople.some((name) => description.includes(name));
    return { description, redFlag };
  });
  const blameless = factors.every((f) => !f.redFlag);
  return { incidentName: incident.name, timeline, contributingFactors: factors, actionItems: incident.actionItems, blameless };
}

const latencyIncident = {
  name: 'recommendations: p95Latency breaks the guardrail at the 10% stage',
  events: [
    { time: '14:20', what: 'The incident is declared (Sev-2 severity, following module 5\'s runbook).' },
    { time: '13:58', what: 'The rollout advances from canary (1%, clean) to the 10% stage.' },
    { time: '16:30', what: 'The technical cause is identified: the call to the recommendation engine is synchronous/blocking and doesn\'t scale to the 10% stage\'s volume.' },
    { time: '14:12', what: 'guardrailWatch() flags HALT at the 10% stage: p95Latency=910ms, ceiling 800ms.' },
    { time: '16:42', what: 'The incident is closed: rollbackDecision() confirms exposure at 0% and the guardrail is no longer at risk.' },
    { time: '14:15', what: 'The team activates the kill switch: recommendationsFlag.enabled flips to false.' },
    { time: '14:45', what: 'Exposure at 0% is confirmed: no new user sees the variant since the kill switch.' },
  ],
  contributingFactors: [
    'The call to the recommendation engine is synchronous and blocking; it wasn\'t designed for the traffic volume of the rollout\'s 10% stage.',
    'The ramp\'s advance criteria (guardrailWatch) didn\'t include a load test equivalent to the next stage\'s volume before advancing.',
    'No cache existed for the most-requested recommendations, so every request recalculated the engine\'s full result.',
  ],
  actionItems: [
    { owner: 'recommendations-team', action: 'Redesign the call to the recommendation engine in async / non-blocking mode.', due: '2026-08-08' },
    { owner: 'platform-sre', action: 'Add a load test equivalent to the next stage\'s volume as part of guardrailWatch()\'s advance criteria.', due: '2026-08-08' },
    { owner: 'recommendations-team', action: 'Implement caching for the most-requested recommendations.', due: '2026-08-15' },
  ],
};

const postmortem = buildPostmortem(latencyIncident);

console.log('=== PART 1: Postmortem -- ' + postmortem.incidentName + ' ===\n');
postmortem.timeline.forEach((e) => console.log(e.time + '  ' + e.what));
console.log('\nContributing factors:');
postmortem.contributingFactors.forEach((f, i) => console.log((i + 1) + '. [' + (f.redFlag ? 'RED FLAG' : 'blameless') + '] ' + f.description));
console.log('\nAction items:');
postmortem.actionItems.forEach((a, i) => console.log((i + 1) + '. (' + a.owner + ', due ' + a.due + ') ' + a.action));
console.log('\nPostmortem blameless: ' + postmortem.blameless);

Part 2 — Iterate: relaunch through the same ramp

With the three action items already closed, reusing rolloutPlan() unchanged from lesson 6:

function rolloutPlan(stages) {
  const results = [];
  let halted = false;
  for (const s of stages) {
    if (halted) { results.push({ ...s, decision: 'NOT_REACHED' }); continue; }
    const decision = s.advanceIf(s.measured) ? 'ADVANCE' : 'HOLD';
    results.push({ ...s, decision });
    if (decision === 'HOLD') halted = true;
  }
  return results;
}

const ceiling = 800;
const relaunchPlan = [
  { percent: 0.01, label: 'canary 1%', measured: { p95Latency: 674 }, advanceIf: (m) => m.p95Latency <= ceiling },
  { percent: 0.10, label: 'rollout 10%', measured: { p95Latency: 738 }, advanceIf: (m) => m.p95Latency <= ceiling },
  { percent: 0.50, label: 'rollout 50%', measured: { p95Latency: 762 }, advanceIf: (m) => m.p95Latency <= ceiling },
  { percent: 1.00, label: 'rollout 100%', measured: { p95Latency: 781 }, advanceIf: (m) => m.p95Latency <= ceiling },
];

console.log('\n=== PART 2: relaunch through the ramp, after the 3 action items ===\n');
const relaunchResults = rolloutPlan(relaunchPlan);
relaunchResults.forEach((s) => {
  console.log(s.label.padEnd(14) + 'p95=' + String(s.measured.p95Latency).padStart(4) + 'ms  -> ' + s.decision);
});
console.log('\nThe full ramp advanced with no HOLD at all: ' + relaunchResults.every((s) => s.decision === 'ADVANCE'));

Part 3 — Measure again: does the lift hold, or was it novelty?

Reusing noveltyCheck() unchanged from lesson 7, on six weeks of checkoutConversion measured after the full relaunch:

function noveltyCheck(weeklyLift) {
  const first = weeklyLift[0];
  const last = weeklyLift[weeklyLift.length - 1];
  const decayPct = Math.round(((first - last) / first) * 1000) / 10;
  const verdict = decayPct > 50 ? 'NOVELTY (fades)' : 'HOLDS (sustained)';
  return { weeklyLift, first, last, decayPct, verdict };
}

// checkoutConversion lift measured week by week, with recommendations
// ALREADY at 100% of the base (six weeks of post-relaunch data).
const recommendationsPostRelaunchLift = [0.191, 0.183, 0.179, 0.185, 0.180, 0.182];

console.log('\n=== PART 3: noveltyCheck on checkoutConversion lift, 6 weeks post-relaunch ===\n');
console.log(noveltyCheck(recommendationsPostRelaunchLift));

What to expect. Running the full file (the three parts in sequence) with Node, the output is exactly this:

=== PART 1: Postmortem -- recommendations: p95Latency breaks the guardrail at the 10% stage ===

13:58  The rollout advances from canary (1%, clean) to the 10% stage.
14:12  guardrailWatch() flags HALT at the 10% stage: p95Latency=910ms, ceiling 800ms.
14:15  The team activates the kill switch: recommendationsFlag.enabled flips to false.
14:20  The incident is declared (Sev-2 severity, following module 5's runbook).
14:45  Exposure at 0% is confirmed: no new user sees the variant since the kill switch.
16:30  The technical cause is identified: the call to the recommendation engine is synchronous/blocking and doesn't scale to the 10% stage's volume.
16:42  The incident is closed: rollbackDecision() confirms exposure at 0% and the guardrail is no longer at risk.

Contributing factors:
1. [blameless] The call to the recommendation engine is synchronous and blocking; it wasn't designed for the traffic volume of the rollout's 10% stage.
2. [blameless] The ramp's advance criteria (guardrailWatch) didn't include a load test equivalent to the next stage's volume before advancing.
3. [blameless] No cache existed for the most-requested recommendations, so every request recalculated the engine's full result.

Action items:
1. (recommendations-team, due 2026-08-08) Redesign the call to the recommendation engine in async / non-blocking mode.
2. (platform-sre, due 2026-08-08) Add a load test equivalent to the next stage's volume as part of guardrailWatch()'s advance criteria.
3. (recommendations-team, due 2026-08-15) Implement caching for the most-requested recommendations.

Postmortem blameless: true

=== PART 2: relaunch through the ramp, after the 3 action items ===

canary 1%     p95= 674ms  -> ADVANCE
rollout 10%   p95= 738ms  -> ADVANCE
rollout 50%   p95= 762ms  -> ADVANCE
rollout 100%  p95= 781ms  -> ADVANCE

The full ramp advanced with no HOLD at all: true

=== PART 3: noveltyCheck on checkoutConversion lift, 6 weeks post-relaunch ===

{
  weeklyLift: [ 0.191, 0.183, 0.179, 0.185, 0.18, 0.182 ],
  first: 0.191,
  last: 0.182,
  decayPct: 4.7,
  verdict: 'HOLDS (sustained)'
}

The result, laid out clean

The three parts together tell a complete, coherent story: the incident was investigated without blaming anyone, the identified cause was real and systemic — confirmed because fixing it resolved the problem in production, not just in theory — and the benefit that originally justified launching recommendations — the +18.75% conversion product-metrics-and-experimentation-guide had rigorously measured over six weeks — reproduces itself, stably, after the relaunch to 100% of the full base too. None of the three parts alone closes the case: a postmortem without verification that the fix worked is just a theory; a ramp that advances without checking the lift only confirms the risk was contained, not that the benefit holds; and a lift measured only once, without the six weeks, couldn't tell a real winner apart from one inflated by initial curiosity.

Common mistakes

Closing the postmortem without connecting its action items to any later data confirming they worked. What happens: the team writes a flawless postmortem, with systemic factors and concrete action items, but never checks again — with data, as in this project's Part 2 — whether those action items actually solved the problem. Why it happens: writing the postmortem feels like the "hard" work already done, and verifying its effectiveness weeks later, once the team's attention has already moved to another priority, is easy to postpone indefinitely. How to spot it: if nobody can show a latency figure from after the action items closed — only the promise that "it's already fixed" — the postmortem produced a plan, but not a confirmation. How to fix it: as in this project, Part 2 isn't optional — a postmortem without later verification is, at best, a well-documented hypothesis, not a solved problem.

Confusing "the ramp advanced cleanly" with "the iteration project is complete." What happens: seeing Part 2's four stages pass with no HOLD, the team reports the case as closed, without ever measuring Part 3's conversion lift. Why it happens: the guardrail (latency) is what caused the original incident, so fixing it feels, emotionally, like solving "the" problem — even though the business reason for launching recommendations in the first place was never latency, it was conversion. How to spot it: if this project's final report includes no checkoutConversion data measured after the relaunch, the question that started this entire guide — is the winner worth it? — is still unanswered. How to fix it: this project's three parts are sequential for a reason — the guardrail (Part 2) protects against the risk, but only measuring the lift (Part 3) confirms the benefit.

Reporting noveltyCheck()'s result without mentioning it's already aligned with the metrics guide's original result. What happens: when presenting decayPct: 4.7 and the HOLDS verdict, someone describes it as a new, surprising discovery, without connecting that this result — stable lift around 18-19% — is exactly what product-metrics-and-experimentation-guide had already predicted by measuring over a full six weeks from the start. Why it happens: each piece of this guide gets experienced, in the moment, as its own discovery, and it's easy to miss when a new result simply confirms an earlier one instead of revealing something different. How to spot it: if the final report doesn't mention the original +18.75% as a point of comparison, the opportunity to show the case's full coherence — from the experiment to the relaunch — is lost. How to fix it: as this project's "The result, laid out clean" table shows, the post-relaunch lift (17.9%-19.1%) falls, week after week, within the range of the original number — a confirmation, not a surprise, and that coherence is, itself, part of the evidence that the winner is real.

Transfer exercises

Unlike the previous lessons' exercises, this project asks you to apply the full method to a case this module has never seen — the real test of whether you learned to close the postmortem-and-iteration cycle, or just memorized recommendations's result.

Exercise 1 — Write the postmortem for a different incident. Mercado's logistics team (from exercise 2 of module 3's project) has its own incident: its estimated-delivery-time algorithm broke the errorRate guardrail (4%) at the rollout 10% stage, with errorRate: 5.2%. The identified cause: the estimated-time model hadn't been tested with multi-package orders at that volume, and its ramp's advance criteria — copied from recommendations's — didn't include a specific test for that order type. Write, in buildPostmortem()'s format (events, contributingFactors, actionItems), this incident's postmortem, with at least 2 contributing factors and 2 action items, without naming any person.

const logisticsIncident = {
  name: 'delivery-estimates: errorRate breaks the guardrail at rollout 10%',
  events: [
    { time: '10:00', what: 'The estimated-time model\'s rollout advances to the 10% stage.' },
    { time: '10:22', what: 'guardrailWatch() flags HALT: errorRate=5.2%, ceiling 4%.' },
    { time: '10:25', what: 'The delivery-estimates flag\'s kill switch is activated.' },
    { time: '13:00', what: 'The cause is identified: the model wasn\'t trained or tested with multi-package orders at this volume.' },
  ],
  contributingFactors: [
    'The estimated-time model didn\'t include enough multi-package order examples in its training data.',
    'The ramp\'s advance criteria, copied from recommendations\'s, didn\'t include a specific test for multi-package orders before advancing stages.',
  ],
  actionItems: [
    { owner: 'logistics-team', action: 'Retrain the model with a representative sample of multi-package orders.', due: '2026-08-20' },
    { owner: 'platform-sre', action: 'Add a segment test (multi-package) to delivery-estimates\'s ramp advance criteria.', due: '2026-08-20' },
  ],
};

Exercise 2 — Simulate the logistics team's iteration. With exercise 1's postmortem already resolved, the logistics team relaunches its ramp. The measured data: canary 1% → errorRate 0.021, rollout 10% → errorRate 0.033, rollout 50% → errorRate 0.038, rollout 100% → errorRate 0.041. With ceiling: 0.04 (4%), mentally simulate rolloutPlan() (or run it in Node) and report at which stage, if any, it stops.

Exercise 3 — Draft the final executive report. Write the message (120-180 words) the recommendations team would send to Mercado's executive team, closing the full case: the original incident, the postmortem, the relaunch, and the confirmation that the lift holds. Include the key numbers from this project's three parts.

Summary and next step

In this mini-project you closed the full learning cycle on the recommendations incident: you wrote the blameless postmortem with buildPostmortem() — a 7-event timeline, 3 systemic factors, 3 action items, all verified as blameless — you confirmed with rolloutPlan() that the relaunch through the same ramp passes all four stages with not a single HOLD, and you confirmed with noveltyCheck() that the conversion lift holds, stable, for six weeks after the relaunch — a decayPct of just 4.7%, far from the novelty threshold.

With this you close module 6: you have the full discipline of the blameless postmortem (what it is, how it's structured, why language matters) and of the ship → measure → learn loop (why launching is the beginning, how to iterate on a winner, how to confirm it holds), applied start to finish on Mercado's full case.

Where you go next. Module 7 applies this same discipline framework — measure before trusting, iterate with evidence — to the modern layer of shipping AI models: migrating a model with shadow mode, measuring its agreement rate with the previous model before switching to it, and the specific postmortem for an AI incident. Module 8, the capstone of the entire guide, is going to ask you to run all seven complete pieces — flag, rollout, monitoring, rollback, postmortem, iteration, and model migration — on this same recommendations launch, closing the full Product Engineering ecosystem.

Resources

• Google SRE Book, Chapter 15, "Postmortem Culture: Learning from Failure" — sre.google/sre-book/postmortem-culture. The full formal reference for the practice this project's Part 1 executed start to finish. In English.
• Eric Ries, The Lean Startup — theleanstartup.com/principles. The original build-measure-learn loop, the conceptual basis for this project's Parts 2 and 3: iterate and remeasure, not launch and forget. In English.
• Ron Kohavi and Stefan Thomke, "The Surprising Power of Online Experiments" (Harvard Business Review) — hbr.org/2017/09/the-surprising-power-of-online-experiments. Additional context on why confirming a result over time — as this project's Part 3 does — is standard industry practice, not an optional step. In English.