Project: decide whether to migrate recommendations from recs-v1 to recs-v2

Description

This module's seven lessons built, piece by piece, the full discipline of migrating a model safely: why a model isn't static code (L2), how to run it in shadow without affecting anyone (L3), how to measure the agreement rate and find where it really differs (L4), how to investigate an incident that doesn't reproduce the same way twice (L5), what data needs logging and what doesn't (L6), and how it all comes together into a formal decision process (L7). This mini-project reuses the three central pieces — shadowCompare(), modelMigrationDecision(), and redactPII() — without changing a single line, on the full case: deciding, with evidence and not intuition, whether recs-v2 is ready for a canary.

Connection to the module. This project doesn't introduce any new function: it reuses buildMercadoShadowTraffic(), recsV1(), and recsV2() from lessons 3 and 4, shadowCompare() from lesson 4, modelMigrationDecision() from lesson 7, and redactPII() from lesson 6 — exactly as they stood, with no change, just as module 4's project reused guardrailWatch() without touching it. What it adds is the discipline of a complete report, bringing together the comparison result, the formal decision, and the definition of what data gets logged, into a single deliverable for Mercado's team.

An analogy: the flight report, before authorizing the next takeoff

Before an airline authorizes a new engine for a flight with passengers, someone has to sign a formal report that brings together all the evidence from previous tests: how many simulator hours it logged, in which maneuvers it matched the usual engine and in which it didn't, what formal decision the safety committee made with those numbers, and what test data gets kept in the permanent archive and what gets discarded as unnecessary. That report isn't a formality — it's what lets anyone, including someone who took part in none of the tests, trust that the decision to authorize (or not authorize) the next flight was made with complete evidence, not with an informal impression of "it looked pretty good."

This project is that report, closed out on recs-v2's real case: from the full shadow traffic to the formal decision and the definition of what data stays in the permanent log.

The complete report, step by step

Part 1 — Run shadowCompare() on the full shadow traffic

We reuse buildMercadoShadowTraffic(), recsV1(), recsV2(), and shadowCompare() exactly as they stood in lessons 3 and 4, on the 50 real shadow-traffic requests.

Part 2 — Isolate the segment where the differences concentrate

From Part 1's result, we confirm the disagreement isn't spread evenly, but almost entirely concentrated in a single segment.

Part 3 — The formal decision with modelMigrationDecision()

We reuse lesson 7's modelMigrationDecision(), unchanged, on Part 1's real result.

Part 4 — What data gets logged from a real case, with redactPII()

We take one of Part 1's real difference records and apply lesson 6's redactPII(), unchanged, to define what makes it into the permanent log.

// PROJECT: decide whether to migrate Mercado's recommendations from recs-v1
// to recs-v2. Reuses buildMercadoShadowTraffic(), recsV1(), recsV2(),
// shadowCompare(), modelMigrationDecision(), and redactPII() EXACTLY as they
// stood in lessons 3, 4, 6, and 7, with no change.

const CATEGORIES = ['electronics', 'home', 'fashion', 'sports', 'grocery', 'beauty', 'toys'];
const GENERAL_TRENDING = 'electronics';
const REGION_TRENDING = { north: 'electronics', south: 'home', east: 'fashion', west: 'sports' };
const REGIONS = ['north', 'south', 'east', 'west'];

function recsV1(request) {
  if (request.hasHistory) return request.lastPurchaseCategory;
  return GENERAL_TRENDING;
}

function recsV2(request) {
  if (request.hasHistory) return request.lastPurchaseCategory;
  return REGION_TRENDING[request.region];
}

function buildMercadoShadowTraffic() {
  const requests = [];
  for (let i = 0; i < 35; i++) {
    requests.push({
      requestId: 'req-hist-' + String(i).padStart(2, '0'),
      segment: 'with-history',
      hasHistory: true,
      lastPurchaseCategory: CATEGORIES[i % CATEGORIES.length],
      region: REGIONS[i % REGIONS.length],
    });
  }
  for (let i = 0; i < 15; i++) {
    requests.push({
      requestId: 'req-cold-' + String(i).padStart(2, '0'),
      segment: 'cold-start',
      hasHistory: false,
      lastPurchaseCategory: null,
      region: REGIONS[i % REGIONS.length],
    });
  }
  return requests;
}

function shadowCompare(oldOutputs, newOutputs) {
  const bySegment = {};
  const differences = [];
  let agreements = 0;
  oldOutputs.forEach((oldReq, i) => {
    const newReq = newOutputs[i];
    const agree = oldReq.topRec === newReq.topRec;
    const seg = oldReq.segment;
    if (!bySegment[seg]) bySegment[seg] = { total: 0, agreements: 0 };
    bySegment[seg].total++;
    if (agree) {
      bySegment[seg].agreements++;
      agreements++;
    } else {
      differences.push({ requestId: oldReq.requestId, segment: seg, oldTopRec: oldReq.topRec, newTopRec: newReq.topRec });
    }
  });
  const total = oldOutputs.length;
  const agreementRate = (agreements / total) * 100;
  const segmentBreakdown = Object.entries(bySegment).map(([segment, s]) => ({
    segment, total: s.total, agreements: s.agreements, agreementRate: (s.agreements / s.total) * 100,
  }));
  return { total, agreements, agreementRate, differences, segmentBreakdown };
}

function modelMigrationDecision({ agreementRate, minAgreement, criticalSegmentAgreementRate, minCriticalSegmentAgreement }) {
  if (agreementRate < minAgreement) {
    return 'DO NOT promote to canary: overall agreement rate is below the minimum -- review the differences first';
  }
  if (criticalSegmentAgreementRate < minCriticalSegmentAgreement) {
    return 'DO NOT promote to canary yet: a critical segment differs too much -- diagnose that strategy before exposing real traffic';
  }
  return 'promote recs-v2 to canary (1%), watching M4\'s guardrails at every stage';
}

function pseudonymizeId(id) {
  let hash = 0;
  for (let i = 0; i < id.length; i++) hash = (hash * 31 + id.charCodeAt(i)) % 100000;
  return 'usr-' + String(hash).padStart(5, '0');
}

function redactPII(record) {
  const FULLY_REMOVE = ['fullName', 'shippingAddress', 'phoneNumber'];
  const MASK = ['email'];
  const clean = { ...record };
  FULLY_REMOVE.forEach((field) => { delete clean[field]; });
  MASK.forEach((field) => {
    if (clean[field]) {
      const [localPart, domain] = clean[field].split('@');
      clean[field] = localPart.slice(0, 2) + '***@' + domain;
    }
  });
  if (clean.userId) clean.userId = pseudonymizeId(clean.userId);
  return clean;
}

// Part 1: run shadowCompare on Mercado's full shadow traffic
console.log('=== Part 1: shadowCompare on the 50 shadow-traffic requests ===\n');
const requests = buildMercadoShadowTraffic();
const oldOutputs = requests.map((r) => ({ requestId: r.requestId, segment: r.segment, topRec: recsV1(r) }));
const newOutputs = requests.map((r) => ({ requestId: r.requestId, segment: r.segment, topRec: recsV2(r) }));
const comparison = shadowCompare(oldOutputs, newOutputs);
console.log('Overall agreement rate: ' + comparison.agreementRate.toFixed(1) + '% (' + comparison.agreements + '/' + comparison.total + ')');
comparison.segmentBreakdown.forEach((s) => {
  console.log('  ' + s.segment.padEnd(14) + 'rate=' + s.agreementRate.toFixed(1) + '%  (' + s.agreements + '/' + s.total + ')');
});

// Part 2: isolate the problematic segment
console.log('\n=== Part 2: the segment where the differences concentrate ===\n');
const coldStart = comparison.segmentBreakdown.find((s) => s.segment === 'cold-start');
console.log('cold-start: ' + coldStart.agreementRate.toFixed(1) + '% agreement, against 100.0% in with-history');
console.log(comparison.differences.length + ' total differences, all in the cold-start segment');

// Part 3: the formal decision with modelMigrationDecision
console.log('\n=== Part 3: decision with modelMigrationDecision ===\n');
const decision = modelMigrationDecision({
  agreementRate: comparison.agreementRate,
  minAgreement: 70,
  criticalSegmentAgreementRate: coldStart.agreementRate,
  minCriticalSegmentAgreement: 60,
});
console.log('Decision: ' + decision);

// Part 4: what gets logged -- redactPII on a real difference case
console.log('\n=== Part 4: what data gets logged from a real case (redactPII) ===\n');
const rawLogCandidate = {
  requestId: 'req-cold-01',
  userId: 'buyer-51190',
  fullName: 'Julio Restrepo Vega',
  email: 'julio.restrepo@example.com',
  shippingAddress: 'Calle 45 #12-30, Medellin',
  phoneNumber: '+57-4-555-0199',
  region: 'south',
  oldTopRec: 'electronics',
  newTopRec: 'home',
  modelVersions: { old: 'recs-v1', new: 'recs-v2' },
  timestamp: '2026-07-22T09:41:55Z',
};
console.log('Record shadow logging wanted to save (with excess PII):');
console.log(JSON.stringify(rawLogCandidate, null, 2));
console.log('\nRecord that actually gets persisted, after redactPII:');
console.log(JSON.stringify(redactPII(rawLogCandidate), null, 2));

What to expect. Running the file with Node, the output is exactly this:

=== Part 1: shadowCompare on the 50 shadow-traffic requests ===

Overall agreement rate: 78.0% (39/50)
  with-history  rate=100.0%  (35/35)
  cold-start    rate=26.7%  (4/15)

=== Part 2: the segment where the differences concentrate ===

cold-start: 26.7% agreement, against 100.0% in with-history
11 total differences, all in the cold-start segment

=== Part 3: decision with modelMigrationDecision ===

Decision: DO NOT promote to canary yet: a critical segment differs too much -- diagnose that strategy before exposing real traffic

=== Part 4: what data gets logged from a real case (redactPII) ===

Record shadow logging wanted to save (with excess PII):
{
  "requestId": "req-cold-01",
  "userId": "buyer-51190",
  "fullName": "Julio Restrepo Vega",
  "email": "julio.restrepo@example.com",
  "shippingAddress": "Calle 45 #12-30, Medellin",
  "phoneNumber": "+57-4-555-0199",
  "region": "south",
  "oldTopRec": "electronics",
  "newTopRec": "home",
  "modelVersions": {
    "old": "recs-v1",
    "new": "recs-v2"
  },
  "timestamp": "2026-07-22T09:41:55Z"
}

Record that actually gets persisted, after redactPII:
{
  "requestId": "req-cold-01",
  "userId": "usr-59282",
  "email": "ju***@example.com",
  "region": "south",
  "oldTopRec": "electronics",
  "newTopRec": "home",
  "modelVersions": {
    "old": "recs-v1",
    "new": "recs-v2"
  },
  "timestamp": "2026-07-22T09:41:55Z"
}

Go over the four parts with what each one confirms. Part 1 runs exactly lesson 4's same shadowCompare(), with no change, on the full shadow traffic: 78.0% overall agreement, with 100.0% concentrated in the with-history segment and just 26.7% at cold-start. Part 2 isolates that result into a sentence anyone can understand without reading code: the 11 total differences are, all of them, in the same segment — there's no disagreement scattered across the rest of the traffic. Part 3 translates that finding into a formal, unambiguous decision: modelMigrationDecision() stops the migration, not because the new model is bad overall, but because a specific, well-identified segment isn't ready yet to be exposed to real buyers. Part 4 closes with the data discipline: out of a record with seven potentially sensitive fields, only two survive whole (region, which isn't PII) and one stays masked (email) — userId gets pseudonymized to usr-59282, and fullName, shippingAddress, and phoneNumber disappear entirely, because none of the three were needed for this log's purpose.

The decision, laid out clean

Check	Result
Overall agreement rate	78.0% (39/50) — clears the 70% threshold
Agreement rate, with-history segment	100.0% (35/35)
Agreement rate, cold-start segment	26.7% (4/15) — below the 60% critical threshold
Total differences	11, all concentrated in cold-start
Formal decision	DO NOT promote to canary yet. Diagnose the cold-start strategy first.
Log data before redactPII()	7 fields, 3 of them PII with no need for the log's purpose
Log data after redactPII()	fullName, shippingAddress, phoneNumber removed; email masked; userId pseudonymized

Notice the table doesn't say "cancel recs-v2" anywhere — it says, precisely, "don't promote yet," exactly the distinction lessons 4 and 7 taught you to hold onto. The new model matches the usual one perfectly for the largest segment of buyers (with history); what this project confirms is that the new cold-start strategy — regional trend instead of overall trend — needs deeper investigation before a single new buyer sees it, not that recs-v2's entire project is scrapped.

Common mistakes

Reporting the decision without the segment breakdown that backs it. What happens: someone summarizes this project as "we're not migrating yet," without mentioning the problem is concentrated at cold-start or that the rest of the traffic is already practically ready. Why it happens: the final conclusion ("don't migrate") feels like the only information that needs communicating, and the detail of which specific segment failed seems like an optional add-on. How to spot it: if the report can't answer "which part of recs-v2 is ready, and which isn't?", half the evidence backing the decision is missing — and without that detail, someone could mistakenly conclude recs-v2 needs to be scrapped entirely. How to fix it: as in this project's Part 2, the complete report needs the exact segment, its two compared agreement rates, and the difference count — not just the final decision's word "DO NOT."

Investigating the cold-start problem without rerunning shadow after the change. What happens: the team adjusts recs-v2's cold-start strategy — based on this project's diagnosis — and, trusting the adjustment "surely fixed it," goes straight to the canary without running shadowCompare() again on the corrected model. Why it happens: after precisely diagnosing a problem, the fix feels like the final step, and repeating the whole measurement cycle feels redundant. How to spot it: if the decision to advance to canary after an adjustment doesn't come with a new shadowCompare() result, the fix was never validated with data, only with the intuition that it should work. How to fix it: any change to the model — including a fix targeted at the failing segment — restarts the cycle: shadow, measure, decide, exactly this project's same process, not an exception for adjustments that "surely already work."

Considering this module's work done without having connected the result to module 8. What happens: the team confirms the decision not to migrate yet and considers the work finished there, without having articulated yet how this same case — the metrics guide's winner with a broken guardrail — ties into everything else the full guide taught. Why it happens: this project's decision feels like the end of the model's story, because it's the question this module set out to answer. How to spot it: if nobody can say "how does this decision about recs-v2 connect to recommendations's already-in-progress rollout?", this module's work ended well, but the guide's full cycle hasn't yet. How to fix it: this project delivers, precisely, the decision about the model's migration — bringing that piece together with the flag, the rollout, the monitoring, the rollback, and the postmortem from the rest of the guide, all on the same Mercado case, is exactly module 8's job, the capstone that closes the full guide.

Transfer exercises

Unlike the previous lessons' exercises, this project asks you to apply the full mechanism to a case this module has never seen — the real test of whether you learned to measure and decide, or just memorized recs-v2's result.

Exercise 1 — Run the project on searchRankerV2. Mercado's search team wants to migrate its search results ranking model, searchRankerV2, and ran its own shadow test: out of 40 total requests, 34 matched the previous model and 6 didn't, all 6 differences concentrated in searches of fewer than 3 words (12 requests of that type in total, of which only 6 matched). Using shadowCompare() conceptually (without rewriting the code, just the reasoning), what would the overall agreement rate be, and what would the short-search segment's be?

See solution

The overall agreement rate would be 34 / 40 * 100 = 85.0%. The short-search segment would have 6 / 12 * 100 = 50.0% agreement (6 of the 12 short searches matched, since the 6 total differences are, per the prompt, concentrated there). With an overall threshold of 70% and a critical segment threshold of 60% — the same ones this project used — modelMigrationDecision() would pass the first check (85.0 > 70) but fail the second (50.0 < 60), returning the same decision recs-v2 got: don't promote yet, the short-search segment needs diagnosis before exposure.

Exercise 2 — Design Part 5: the message to the product team. Write the message (100-150 words) you'd send Mercado's product team, reporting the decision not to promote recs-v2 to canary yet. Include: the overall result, the problematic segment with its number, that the rest of the model is ready, and what's next (without explaining the technical detail of the fix yet, just naming that the team is going to diagnose it).

See solution

One possible message: "We finished running recs-v2 in shadow on real Mercado traffic: it matches recs-v1 in 78% of cases. But that number hides something important — for buyers with purchase history, agreement is perfect (100%); for new buyers with no history, it drops to just 27%. The 11 differences we found are all concentrated in that second group: recs-v2 changed how it recommends to new buyers, using regional trends instead of the site's overall trend. We still don't know if that change is an improvement or a problem, so we're not going to turn on the canary until we investigate it. The rest of the model — most of the traffic — is already ready. The team is going to specifically diagnose the cold-start strategy before the next round of shadow." The message clearly separates what's ready from what isn't, gives the exact numbers for both segments, and makes explicit that the decision is "not yet," not "never."

Exercise 3 — Prediction about module 8. Without having read module 8 yet, and based on everything you built in this project and the rest of the guide, what do you think the final capstone is going to need to bring together on recommendations's case, beyond this decision about the model?

See solution

The capstone will probably need to bring together, on recommendations's same case, every piece the full guide built: the flag behind which the feature lives (module 2), the gradual rollout plan with its stages (module 3), the guardrail monitoring at every stage — including the latency that broke at 10% (module 4) —, the decision to stop or revert facing that broken guardrail (module 5), that regression's blameless postmortem (module 6), and now, the decision about whether the model powering the entire feature is ready for its own migration (this module 7). The capstone, then, probably introduces no new concept — it brings together, on a single complete case, every formal decision the guide taught you to make separately.

Summary and next step

In this mini-project you brought together the full migration process on recs-v2's real case: shadowCompare() confirmed 78.0% overall agreement with a critical 26.7% at the cold-start segment, modelMigrationDecision() translated that result into a formal decision not to promote yet, and redactPII() defined exactly what data from a real record — seven fields, four with some degree of personal information — survives the logging process: two whole, one masked, one pseudonymized, three removed. With this you close module 7: you have, in executed and verified code, the complete discipline of comparing before exposing, investigating a probabilistic incident with the right criteria, and logging only what's needed.

Where you go next. Module 8, the full guide's final capstone, brings together the seven pieces built across the previous modules — feature flags, gradual rollout, guardrail monitoring, rollback and incident response, blameless postmortems, and now a model's safe migration — on recommendations's same case, start to finish: the feature behind a flag, its gradual rollout, the latency that breaks the guardrail at 10%, the decision to stop, that regression's postmortem, and the improvement plan for the model powering the whole feature. With that, you close not just this module, but the entire Product Engineering ecosystem — and you know, with evidence and not intuition, how to launch a product change, monitor it, revert it if needed, learn from the result, and safely migrate the AI model behind it.

Resources

• Amazon SageMaker AI, "Shadow tests" — docs.aws.amazon.com/sagemaker/latest/dg/shadow-tests.html. Closes the module where lesson 3 opened it: how a real platform automates exactly the mechanism this project ran by hand, from the shadow comparison to the decision to promote to production. In English.
• Google SRE Workbook, Chapter 16, "Canarying Releases" — sre.google/workbook/canarying-releases. The chapter that formalizes the step that follows a positive modelMigrationDecision(): exposing a small fraction first, and only then continuing — modules 2 and 3's same gradual rollout, now applied to a model. In English.