Module 8: Project Ship Mercados Recommendations

Iterating: migrating the model with shadow mode

Next →

Description

The previous lesson's postmortem left a concrete action item: migrate the recommendation engine from v1 — synchronous, no timeout, the real source of the latency regression — to v2, a faster inference architecture. But "faster" isn't the same as "safe to launch blindly" — and this v2 doesn't arrive at this lesson without history: it's the second iteration of the same model module 7 left in shadow. That first round — shadowCompare() on 50 real requests — found 78.0% overall agreement, but only 26.7% at the cold-start segment, far below the 60% critical threshold modelMigrationDecision() requires: the model was not migrated then. The team diagnosed exactly that cold-start strategy and fixed it before proposing this v2 again. This lesson runs that corrected version in shadow mode once more — in parallel, without affecting any real buyer — and applies, without changing a line, the same shadowCompare() and the same modelMigrationDecision() module 7 built, to confirm whether cold-start really got fixed this time.

Connection to the module. This lesson reuses, without changing a line, shadowCompare() and modelMigrationDecision() exactly as they stood in module 7's lessons 4 and 7 — the two central decision functions that keep a high overall agreement from hiding a broken segment — and builds redactPII(), this capstone's integration piece for the log privacy that lesson 6 of that same module describes. This lesson's result — is it safe to migrate, with the critical segment now fixed? — is the condition lesson 7 needs met before relaunching recommendations.

An analogy: the trainee pilot, back after reviewing the maneuver that failed

Before a pilot in training takes the controls of a real commercial flight, they spend hours in the co-pilot's seat, watching and predicting what they'd do in each situation — without touching anything — while the experienced pilot actually flies. The first time the captain reviewed those predictions, they found something specific: on landings with new passengers on board, the trainee pilot got it wrong most of the time. They didn't authorize them to take the controls — they sent them to review exactly that maneuver.

This lesson is the second review, after that specific training. The trainee pilot — v2 — goes back to watching in parallel with the captain — v1, still genuinely in charge — and the captain reviews their predictions again with the exact same criteria as the first time: not just "do they match most of the time?", but "do they also match, specifically, on the maneuver that failed before?" If the answer to both questions is yes, that's real evidence the review worked. If that specific maneuver still fails, it doesn't matter how good the overall average looks — it still isn't time to hand over control.

Worked example: shadowCompare() and modelMigrationDecision(), reused from M7, on v2's second iteration

// M8 L06: runs v2's SECOND iteration in shadow (with cold-start already fixed)
// and applies, without changing a line, shadowCompare() (M7 L4) and
// modelMigrationDecision() (M7 L7) -- the same pair of functions that caught
// the cold-start problem the first time. redactPII() protects the
// comparison log's personal data before saving it -- the privacy practice
// module 7 describes for shipping AI safely.

// shadowCompare: EXACTLY M7 L4's segment-aware version -- overall agreement rate
// PLUS a segment breakdown, so a high average doesn't hide a broken segment.
function shadowCompare(oldOutputs, newOutputs) {
  const bySegment = {};
  const differences = [];
  let agreements = 0;
  oldOutputs.forEach((oldReq, i) => {
    const newReq = newOutputs[i];
    const agree = oldReq.topRec === newReq.topRec;
    const seg = oldReq.segment;
    if (!bySegment[seg]) bySegment[seg] = { total: 0, agreements: 0 };
    bySegment[seg].total++;
    if (agree) {
      bySegment[seg].agreements++;
      agreements++;
    } else {
      differences.push({ requestId: oldReq.requestId, segment: seg, oldTopRec: oldReq.topRec, newTopRec: newReq.topRec });
    }
  });
  const total = oldOutputs.length;
  const agreementRate = (agreements / total) * 100;
  const segmentBreakdown = Object.entries(bySegment).map(([segment, s]) => ({
    segment, total: s.total, agreements: s.agreements, agreementRate: (s.agreements / s.total) * 100,
  }));
  return { total, agreements, agreementRate, differences, segmentBreakdown };
}

// modelMigrationDecision: EXACTLY M7 L7's version -- checks TWO thresholds,
// overall AND the most critical segment's, before authorizing a canary.
function modelMigrationDecision({ agreementRate, minAgreement, criticalSegmentAgreementRate, minCriticalSegmentAgreement }) {
  if (agreementRate < minAgreement) {
    return 'DO NOT promote to canary: overall agreement rate is below the minimum -- review the differences first';
  }
  if (criticalSegmentAgreementRate < minCriticalSegmentAgreement) {
    return 'DO NOT promote to canary yet: a critical segment differs too much -- diagnose that strategy before exposing real traffic';
  }
  return 'promote recs-v2 to canary (1%), watching M4\'s guardrails at every stage';
}

function redactPII(logEntry, fields) {
  const redacted = { ...logEntry };
  fields.forEach((f) => { if (redacted[f] !== undefined) redacted[f] = '[REDACTED]'; });
  return redacted;
}

// buildShadowTraffic: 20 real shadow requests -- 7 buyers with history
// (with-history) and 13 new buyers (cold-start), the segment v2's first
// iteration (module 7) left unresolved.
function buildShadowTraffic() {
  const requests = [];
  for (let i = 0; i < 20; i++) {
    requests.push({ requestId: 'buyer-' + i, segment: i < 7 ? 'with-history' : 'cold-start' });
  }
  return requests;
}

console.log('=== Part 1: shadowCompare -- v2 (iteration 2, cold-start fixed) runs in parallel ===\n');
const requests = buildShadowTraffic();
const oldTopRecs = ['sku-104','sku-221','sku-330','sku-104','sku-558','sku-221','sku-777','sku-104','sku-330','sku-909',
  'sku-221','sku-104','sku-558','sku-330','sku-777','sku-221','sku-104','sku-909','sku-330','sku-558'];
const newTopRecs = ['sku-104','sku-221','sku-330','sku-104','sku-558','sku-221','sku-777','sku-441','sku-330','sku-909',
  'sku-221','sku-104','sku-558','sku-330','sku-616','sku-221','sku-104','sku-909','sku-330','sku-558'];
const oldOutputs = requests.map((r, i) => ({ requestId: r.requestId, segment: r.segment, topRec: oldTopRecs[i] }));
const newOutputs = requests.map((r, i) => ({ requestId: r.requestId, segment: r.segment, topRec: newTopRecs[i] }));
const shadow = shadowCompare(oldOutputs, newOutputs);
console.log('overall agreementRate=' + shadow.agreementRate.toFixed(1) + '% (' + shadow.agreements + '/' + shadow.total + ')');
shadow.segmentBreakdown.forEach((s) => {
  console.log('  ' + s.segment.padEnd(14) + 'rate=' + s.agreementRate.toFixed(1) + '%  (' + s.agreements + '/' + s.total + ')');
});
console.log('diffs: ' + shadow.differences.map((d) => d.requestId + ' [' + d.segment + ']: ' + d.oldTopRec + ' -> ' + d.newTopRec).join(', '));

console.log('\n=== Part 2: modelMigrationDecision -- the same double threshold that caught the problem in M7 ===\n');
const coldStart = shadow.segmentBreakdown.find((s) => s.segment === 'cold-start');
const decision = modelMigrationDecision({
  agreementRate: shadow.agreementRate,
  minAgreement: 70,
  criticalSegmentAgreementRate: coldStart.agreementRate,
  minCriticalSegmentAgreement: 60,
});
console.log('overall agreementRate=' + shadow.agreementRate.toFixed(1) + '%  (minimum=70%)');
console.log('cold-start agreementRate=' + coldStart.agreementRate.toFixed(1) + '%  (minimum=60%)');
console.log('Decision: ' + decision);

console.log('\n=== Part 3: redactPII on the shadow run\'s log, before saving it ===\n');
const rawLogEntry = { buyerId: 'buyer-00014', email: 'ana@example.com', ip: '190.12.44.8', oldRec: 'sku-777', newRec: 'sku-616' };
const cleanLogEntry = redactPII(rawLogEntry, ['email', 'ip']);
console.log('raw:       ' + JSON.stringify(rawLogEntry));
console.log('redacted:  ' + JSON.stringify(cleanLogEntry));

What to expect. Running the file with Node, the output is exactly this:

=== Part 1: shadowCompare -- v2 (iteration 2, cold-start fixed) runs in parallel ===

overall agreementRate=90.0% (18/20)
  with-history  rate=100.0%  (7/7)
  cold-start    rate=84.6%  (11/13)
diffs: buyer-7 [cold-start]: sku-104 -> sku-441, buyer-14 [cold-start]: sku-777 -> sku-616

=== Part 2: modelMigrationDecision -- the same double threshold that caught the problem in M7 ===

overall agreementRate=90.0%  (minimum=70%)
cold-start agreementRate=84.6%  (minimum=60%)
Decision: promote recs-v2 to canary (1%), watching M4's guardrails at every stage

=== Part 3: redactPII on the shadow run's log, before saving it ===

raw:       {"buyerId":"buyer-00014","email":"ana@example.com","ip":"190.12.44.8","oldRec":"sku-777","newRec":"sku-616"}
redacted:  {"buyerId":"buyer-00014","email":"[REDACTED]","ip":"[REDACTED]","oldRec":"sku-777","newRec":"sku-616"}

Part 1 runs shadowCompare() on 20 sample buyers, already split into the two segments that matter: 18 of 20 match overall (90.0% overall), but the breakdown is what really answers this lesson's question — with-history reaches 100.0% (7/7), and cold-start, the segment v2's first iteration left at 26.7% in module 7, now reaches 84.6% (11/13). The two remaining differences — buyer-7 and buyer-14 — are both concentrated in cold-start: they didn't disappear entirely, but the improvement is huge, and they no longer drag the segment below any reasonable threshold.

Part 2 applies modelMigrationDecision(), without changing a single letter from module 7, to this result. The first check (90.0 < 70) is false, so it passes. The second check (84.6 < 60) is also false — unlike the first iteration, where 26.7 < 60 was true and stopped the migration right there. With both thresholds cleared, the function returns the authorization: 'promote recs-v2 to canary (1%), watching M4\'s guardrails at every stage'. Notice the message still says recs-v2 — it's literally the same string module 7's function returns, with no change, because it's the same function; Mercado uses v2 as a shorthand for the same model in this capstone's everyday conversation.

Part 3 shows the privacy piece: before saving this comparison's log to any analysis or debugging system, redactPII() replaces the fields identifying a specific person (email, ip) with [REDACTED], leaving intact the fields actually needed for the technical analysis (buyerId as an opaque identifier, and the recommendations themselves). Shadow mode watches the model's behavior, it doesn't need, and shouldn't save, the buyer's personal data to do that.

Why the migration needs TWO thresholds, not one — and why it clears them this time

This result's most important lesson isn't the 90.0% number — it's that that number, alone, would never have been enough to decide anything. modelMigrationDecision(), the exact same function module 7 built, doesn't ask "does overall agreement clear a threshold?" and stop there — it checks two conditions, in order: first overall agreement against a minimum (70%), and only if that passes, the most critical segment's agreement against its own minimum (60%). This second v2 iteration passes both: 90.0% overall, well above 70%, and 84.6% at cold-start, comfortably above 60% — a huge improvement over the 26.7% the first iteration left in shadow in module 7, far below that same critical threshold.

Notice what this confirms about the process, beyond the specific result: modelMigrationDecision() didn't change a single line between module 7 and this lesson — it was the model that changed, after the team specifically diagnosed the segment that was failing and fixed it. That's exactly the cycle module 6 (ship → measure → learn) taught you to apply to any result: measure rigorously, find where something an average hides is actually failing, fix that specific cause, and remeasure with the same yardstick — not a more permissive one — before trusting the new result.

Common mistakes

Migrating straight to production trusting "the second iteration surely already fixed it," without rerunning shadow. What happens: after fixing the cold-start strategy, someone proposes jumping straight to the canary, reasoning the specific problem was already identified and fixed, so repeating the full shadow test feels redundant. Why it happens: having diagnosed the exact cause the first time generates confidence — sometimes excessive — that the fix worked exactly as expected, without verifying it with new data. How to spot it: if there's no second shadowCompare() result, after the change, with its own segment breakdown, the improvement is a hypothesis, not a confirmed fact. How to fix it: as in this lesson's Parts 1 and 2, any change to the model — including a targeted fix — restarts the full cycle: shadow, measure with the same breakdown, decide with the same two thresholds.

Reporting only the 90.0% overall agreement, without the segment breakdown that confirms cold-start really got fixed. What happens: someone summarizes this lesson's result as "90% agreement, much better than last time," without citing the cold-start segment's specific 84.6%. Why it happens: after already having gone through the surprise of a hidden segment in module 7, it's tempting to assume that, this time, the overall number already tells the whole story. How to spot it: if nobody in the report can cite the cold-start segment's agreement rate specifically, the verification landed in the same blind spot that almost authorized an unreviewed migration in module 7. How to fix it: always report the full breakdown alongside the overall number — exactly as this lesson's Part 1 does — no matter how much the aggregate improved.

Saving the shadow run's logs without passing them through redactPII(), "because it's just for internal debugging." What happens: someone on the team saves the shadow run's full differences — including buyers' email and ip — into a logging system the whole team can query, with nothing redacted, arguing "it's just for internal use, there's no risk." Why it happens: shadow mode's data feels "less sensitive" than production data, because no user sees the result — but the personal data entering the comparison is still real data about real people. How to spot it: if any shadow run log contains an unredacted email or IP address, this lesson's privacy practice wasn't applied. How to fix it: as in this lesson's Part 3, any log saved from the shadow run goes through redactPII() before being written, no exceptions, even when the purpose is exclusively technical.

Exercises

Exercise 1 — Calculate the verdict with a stricter segment threshold. Mercado's security team, more conservative after module 7's incident, proposes raising the critical segment threshold from 60% to 90% for this second iteration. With this lesson's same data (agreementRate=90.0, cold-start=84.6), what decision would modelMigrationDecision() return?

See solution

'DO NOT promote to canary yet...'. The first check (90.0 < 70) is still false, so it passes. But the second check, with minCriticalSegmentAgreement: 90, becomes 84.6 < 90, which is true — so the function stops there, without authorizing the canary, despite 84.6% being a huge improvement over the original 26.7%. This exercise confirms the same idea as module 7's equivalent exercise: the threshold isn't an objective value shadowCompare() calculates — it's a business decision, and raising it can change the final decision even when the model genuinely improved.

Exercise 2 — Design a log for another sensitive field. The team discovers the sessionDeviceFingerprint field (a technical identifier for the buyer's device) should also be redacted in the shadow run's logs, because it can be used to identify a specific person when combined with other data. How would you call redactPII() to redact email, ip, and sessionDeviceFingerprint at once, on this lesson's same rawLogEntry (assuming that field also existed in the object)?

See solution

redactPII(rawLogEntry, ['email', 'ip', 'sessionDeviceFingerprint']). The function is already designed to accept any list of fields to redact — the second parameter, fields, is an array redactPII() walks with forEach, with no limit on its size — so adding a third sensitive field requires no change to the function's code, only to the list passed to it. This is, in miniature, the same design principle behind this entire guide: functions receive configuration as data, not as hardcoded logic.

Exercise 3 — Explain shadow mode without technical jargon. In 2-3 sentences, without using the words "shadow," "model," or "agreement rate," explain to someone from the business side why the team didn't switch straight to the new recommendation engine, but instead first had it run "in secret" for a few days, for a second time.

See solution

One example answer: "Before using the new recommendation system with real buyers, we ran it in parallel with the current system, without anyone seeing it, just to compare how similar its suggestions were. The first time we ran this test, we found it failed specifically with new buyers who don't have purchase history yet, so we didn't launch it. The team specifically fixed that part, and this second test confirms it now matches 9 out of every 10 cases overall, and also matches consistently with that group of new buyers that used to be the problem."

Summary and next step

In this lesson you validated, with shadowCompare() and modelMigrationDecision() reused unchanged from module 7, that the recommendation model's second v2 iteration is safe to migrate: 90.0% overall agreement and 84.6% at the cold-start segment — the same segment that left 26.7% in module 7's first shadow round — both above their thresholds (70% and 60% respectively). You also confirmed, with redactPII(), that this comparison's logs protect buyers' personal data before being saved.

Before moving on you should be able to: explain why modelMigrationDecision() needs to check two thresholds, not one; and name which fields of a comparison log would need redacting before being saved.

With the latency fix (the faster v2 engine) and cold-start now fixed and validated, lesson 7 relaunches recommendations through module 3's same ramp — and verifies something no previous lesson could confirm yet: whether the +18.75% lift holds weeks after the relaunch, or whether it was just the novelty effect.

Resources

← PreviousThe postmortem, blaming no oneNext →Relaunching, and verifying the lift holds