Module 7: Shipping Ai Safely

Migrating a model end to end: from shadow to production, no shortcuts

Next →

Description

The previous five lessons built pieces: serveWithShadow() (L3), shadowCompare() with its agreement rate and segment breakdown (L4), checkReproducibility() for investigating probabilistic incidents (L5), redactPII() for not logging more than necessary (L6). This lesson brings them together into a complete process, start to finish, and adds the missing piece: modelMigrationDecision(), the function that turns shadowCompare()'s result into an explicit decision about whether or not to promote a model to canary — the same formal-decision discipline launchVerdict() (M1) and rollbackDecision() (M5) already applied to other decisions in this guide.

Connection to the module. This lesson doesn't replace anything built before — it puts it in order. It reuses lesson 4's shadowCompare() exactly as it stood, on exactly the same 50 shadow-traffic requests, and adds the missing decision function so that result translates into a concrete action. Lesson 8's project reuses this exact function, unchanged, on recommendations's full case.

An analogy: the day of the first real leg

Lessons 3 and 4's trainee pilot already has enough shadow hours logged to have a number: they match the captain on 78% of observed maneuvers, with one important detail — on takeoffs and landings with new passengers on board (a flight's "cold-start," in a sense), that match drops to under 27%. A captain who only looked at the 78% might feel tempted to hand over a full leg. A captain who reviewed the breakdown knows it isn't the moment yet — not because the trainee pilot is bad, but because there's a specific type of maneuver that needs more practice before trying it with real passengers.

This lesson is the full protocol that decides when that day arrives, and what happens once it does. It isn't just "the number clears the threshold, go ahead" — it's a full sequence: confirm the number, confirm no critical segment is lagging behind, hand over only a short leg at first (modules 2 and 3's canary), keep the captain with a hand ready to retake control (module 2's kill switch), watch the instruments through the whole leg (module 4's guardrails), and have it clear, in advance, what to do if something goes wrong (module 5's rollback).

Worked example: modelMigrationDecision() on shadowCompare()'s result

Let's build the decision function and apply it to the exact result shadowCompare() produced in lesson 4 — 78.0% overall agreement, 26.7% at the cold-start segment:

// modelMigrationDecision: checks shadowCompare()'s result against TWO
// explicit thresholds -- overall agreement, AND the most critical segment's
// agreement -- before authorizing any canary. A single overall threshold
// isn't enough (lesson 4): one segment can be far below without the overall
// number revealing it.
function modelMigrationDecision({ agreementRate, minAgreement, criticalSegmentAgreementRate, minCriticalSegmentAgreement }) {
  if (agreementRate < minAgreement) {
    return 'DO NOT promote to canary: overall agreement rate is below the minimum -- review the differences first';
  }
  if (criticalSegmentAgreementRate < minCriticalSegmentAgreement) {
    return 'DO NOT promote to canary yet: a critical segment differs too much -- diagnose that strategy before exposing real traffic';
  }
  return 'promote recs-v2 to canary (1%), watching M4\'s guardrails at every stage';
}

// shadowCompare()'s real result on Mercado's case, from lesson 4
// (reproduced here as input data; the full function lives in L4).
const shadowResult = {
  agreementRate: 78.0,
  segmentBreakdown: [
    { segment: 'with-history', agreementRate: 100.0 },
    { segment: 'cold-start', agreementRate: 26.7 },
  ],
};

console.log('=== modelMigrationDecision on shadowCompare\'s result ===\n');
const coldStartSegment = shadowResult.segmentBreakdown.find((s) => s.segment === 'cold-start');
const decision = modelMigrationDecision({
  agreementRate: shadowResult.agreementRate,
  minAgreement: 70,
  criticalSegmentAgreementRate: coldStartSegment.agreementRate,
  minCriticalSegmentAgreement: 60,
});
console.log('overall agreementRate=' + shadowResult.agreementRate.toFixed(1) + '%  (minimum=70%)');
console.log('cold-start agreementRate=' + coldStartSegment.agreementRate.toFixed(1) + '%  (minimum=60%)');
console.log('Decision: ' + decision);

What to expect. Running the file with Node, the output is exactly this:

=== modelMigrationDecision on shadowCompare's result ===

overall agreementRate=78.0%  (minimum=70%)
cold-start agreementRate=26.7%  (minimum=60%)
Decision: DO NOT promote to canary yet: a critical segment differs too much -- diagnose that strategy before exposing real traffic

The overall 78.0% does pass the first threshold (70%) — if modelMigrationDecision() only looked at that number, the decision would be "go ahead, canary." But the second check, on the cold-start segment, finds a 26.7% far below the 60% critical threshold — and that single signal is enough to stop the entire decision, no matter how good the overall number looks. Notice the order of the two ifs: overall first, then the critical segment — if the overall one had already failed, there wouldn't even be a need to check the segment, because there'd already be no basis for migrating. This is exactly the same staged-decision pattern launchVerdict() (M1) and rollbackDecision() (M5) already used: several explicit conditions, evaluated in an order that makes sense, instead of a single question that collapses all the evidence into one yes or no.

The full process, start to finish

With modelMigrationDecision() as the hinge, here's what the full process of migrating recs-v1 to recs-v2 looks like, joining this module's pieces with the previous modules' pieces:

1. SHADOW MODE (L3)
   recs-v2 runs in parallel to recs-v1, on real traffic.
   Zero impact on what the buyer sees.
        │
        ▼
2. MEASURE (L4)
   shadowCompare() over enough accumulated volume.
   Overall agreement rate + segment breakdown.
        │
        ▼
3. DECIDE (L7 -- this lesson)
   modelMigrationDecision(): do the overall rate AND
   every critical segment pass their threshold?
        │
   ┌────┴────┐
   NO         YES
   │           │
   ▼           ▼
Diagnose      4. CANARY (M2 + M3)
the failing   recs-v2 behind a feature flag, gradual
segment,      rollout: 1% → 10% → 50% → 100%, with
back to       isEnabled() deciding who sees which model.
shadow              │
                    ▼
              5. WATCH (M4)
              guardrailWatch() at every stage: latency,
              complaints, churn, margins -- the usual ones.
                    │
              ┌─────┴─────┐
              OK           Guardrail broken
              │             │
              ▼             ▼
        Ramp keeps       6. KILL SWITCH + ROLLBACK (M2 + M5)
        climbing         Shuts off recs-v2 instantly, back
                          to recs-v1. rollbackDecision()
                          decides revert or fix-forward.
                                │
                                ▼
                          7. POSTMORTEM (M6 + L5)
                          If the cause is probabilistic,
                          checkReproducibility() first.
                                │
                                ▼
                          8. THE WHOLE PROCESS LOGS
                          ONLY WHAT'S NEEDED (L6)
                          redactPII() on every log generated.

It's worth noting what this diagram confirms: no step in this module replaces modules 2 through 6 — it extends them. A model's canary uses the same isEnabled() and the same rolloutPlan() as any other feature; the kill switch is module 2 lesson 5's same binary mechanism; the rollback is module 5's same rollbackDecision(). The only genuinely new thing is what happens before the flag even exists — shadow mode and the agreement-based decision — and what changes when something goes wrong — the reproducibility question before the postmortem, and the extra care about what gets logged at every step.

Common mistakes

Skipping the decision step, and going straight from "we ran shadow" to "let's turn on the canary." What happens: the team runs recs-v2 in shadow for a reasonable time, looks at shadowCompare()'s result informally — "it looks pretty similar" —, and activates the canary without having applied any explicit threshold or reviewed the segment breakdown. Why it happens: after investing the effort of building the shadow, moving to the canary feels like the natural next step, and an informal read of the result seems enough when the overall number looks reasonable. How to spot it: if nobody can point to an explicit threshold — for overall agreement or for any segment — that was formally compared before activating the canary, the decision was informal, not criteria-based. How to fix it: as in this lesson's example, modelMigrationDecision() with explicit thresholds, defined in advance, is the mandatory step between "we measured" and "we expose" — never a direct jump.

Migrating 100% at once, because "the shadow test already passed." What happens: once modelMigrationDecision() (or its informal equivalent) gives the green light, the team interprets that as authorization to put recs-v2 at 100% immediately, skipping the canary and gradual-ramp stages. Why it happens: passing the shadow test feels like the complete, definitive validation, and it's easy to forget the shadow, by design, hasn't exposed a single real buyer yet — the gradual rollout is still needed afterward, exactly as for any other feature. How to spot it: if the migration plan doesn't include a 1% canary or intermediate stages, it jumped straight from "shadow approved" to "full production," without the rollout modules 2 and 3 already built for this exact purpose. How to fix it: passing the shadow test is the authorization to start the gradual rollout — the same usual canary 1% → 10% → 50% → 100% — not to skip it. This lesson's diagram shows it in order: shadow and decision first, canary and ramp after, never reversed or combined.

Not having a model-specific kill switch, separate from the full feature's flag. What happens: recommendations as a feature has its kill switch (module 2), but the team assumes that same switch works for going back from recs-v2 to recs-v1 — without actually having built a way to switch models without shutting off the entire feature. Why it happens: a single boolean flag for the whole feature feels sufficient, and it's easy to miss the difference between "shutting off recommendations entirely" and "going back to the previous model, keeping the feature on" until the moment that second option is genuinely needed. How to spot it: if the only way to stop using recs-v2 is to shut off recommendations entirely for every buyer, a finer level of control is missing. How to fix it: module 2 lesson 3's flag registry already naturally supports this — an additional field indicating which model version the active flag serves, changeable independently of the general enabled — so a regression in the new model doesn't force shutting off the whole feature to go back to the known version.

Exercises

Exercise 1 — Change the thresholds. If Mercado decided that, for this specific case, the critical segment threshold should be 20 instead of 60 (accepting more difference at cold-start before blocking the migration), what decision would modelMigrationDecision() return with this lesson's same data (agreementRate=78.0, cold-start=26.7)?

See solution

With minCriticalSegmentAgreement: 20, the check 26.7 < 20 is false, so that if no longer blocks the decision. Since the first check (78.0 < 70) is also false, the function would reach the end and return 'promote recs-v2 to canary (1%), watching M4\'s guardrails at every stage'. This exercise shows something important: the threshold isn't an objective value shadowCompare() calculates — it's a business decision about how much difference the team is willing to tolerate before investigating, and changing that number changes the final decision without any real data having changed.

Exercise 2 — Place each piece in the diagram. Without looking at this lesson's diagram, try to recall: at which step of the full process does lesson 5's checkReproducibility() show up, and why at that point and not earlier?

See solution

checkReproducibility() shows up at the postmortem step, after an incident detected during the canary or the rollout — not earlier, because there's no incident to investigate yet during shadow or the migration decision. It makes sense for it to show up right before the postmortem (step 7 of the diagram): before writing down any incident's root cause that involves the model, you first have to confirm whether the responsible component is deterministic or probabilistic, exactly the question lesson 5 teaches you to ask first.

Exercise 3 — Explain the boundary with M2 and M3. A teammate, after reading this module, comments: "so migrating a model is a completely different process from launching a normal feature." Do you agree? Correct the statement using what you learned in this lesson.

See solution

That's not quite right. Migrating a model reuses almost all of launching a normal feature's process — the same feature flag, the same gradual rollout with a canary, the same guardrails, the same rollback — with no change. What's different, and what this module adds, is what happens before the flag even gets created: shadow mode and the agreement-rate-based decision, which have no equivalent in a normal code feature, because a code feature doesn't have a "previous version" to compare its behavior against in the same probabilistic way two models can be compared. The fair correction would be: "migrating a model adds a new layer before the usual process, it doesn't replace the usual process."

Summary and next step

In this lesson you built modelMigrationDecision() and confirmed, with recommendations's real data, that the 78.0% overall agreement isn't enough on its own: the 26.7% at the cold-start segment blocks the migration until that segment gets investigated. You saw the full process start to finish — shadow, measure, decide, canary, watch, kill switch and rollback if something goes wrong, postmortem with the reproducibility question first, and data minimization on every log generated — and confirmed this module doesn't replace anything built in modules 2 through 6: it extends them with what's specific to a model.

Before moving on you should be able to: explain why a migration decision needs two types of threshold, not one; draw the full process from memory, pointing out what's new in this module and what's reused from previous ones; and explain the difference between a full feature's kill switch and a specific model version's.

Lesson 8, this module's mini-project and closing lesson, puts you in Mercado's team's place: running the full process on the real case, reusing shadowCompare(), modelMigrationDecision(), and redactPII() without changing a line, to produce the formal decision and the data report the team needs before touching recs-v2 again.

Resources

← PreviousPrivacy and data when shipping: what gets logged, and what didn't need toNext →Project: decide whether to migrate recommendations from recs-v1 to recs-v2