Shadow mode: running the new model without anyone noticing

Description

Lesson 2 left a question open: when Mercado decides to replace recs-v1 with recs-v2, how do you know if the new model decides differently from the old one, before that difference reaches a single buyer? This lesson builds the answer: shadow mode — running recs-v2 in parallel to recs-v1, on the same real traffic, with none of its decisions reaching the user yet. The buyer keeps seeing, at all times, exactly what recs-v1 decides. recs-v2 runs alongside it, quietly, and its responses get logged for comparison later.

Connection to the module. This lesson builds the infrastructure piece lesson 4 is going to put to real work: today you define recsV1() and recsV2() — the two functions that are going to run in parallel for the rest of the module — and serveWithShadow(), which calls both but only lets the first one's response through. Lesson 4 reuses these exact same two functions, without changing a line, to build shadowCompare() on a real volume of traffic.

An analogy: the trainee pilot, in the seat next door

Before a pilot in training takes the controls of a flight with passengers, they spend many hours in the co-pilot's seat on real flights, with the captain in command: they watch the instruments, make their own mental decisions about what they'd do at each maneuver — when to adjust altitude, when to correct course —, and write them down. The captain never hands over control during those hours. The passengers never know that, in the seat next door, someone is "flying" the same flight in their head, making their own decisions without any of them moving a single real control.

That's shadow mode. recs-v1 is the captain: its decisions are the ones that actually reach the buyer. recs-v2 is the trainee pilot: it receives exactly the same information as recs-v1 — the same request, the same buyer, the same moment —, decides what it would recommend, and that decision gets logged for review, but never touches the real controls. Only after enough "flights" in this mode, with enough decisions logged and compared, does it make sense — lesson 4 onward — to ask whether the trainee pilot is ready for a first real, short, supervised leg: the canary.

Worked example: serveWithShadow() on four Mercado requests

Let's build recsV1() and recsV2(), the two models that are going to run in parallel for the rest of this module, and serveWithShadow(), the function that calls both but only lets the first one's response through to the buyer:

// recsV1 (the production model, the captain): if the buyer has purchase
// history, recommends their last purchased category. If it's a new buyer
// (cold start, no history), recommends the site's overall best-selling category.
function recsV1(request) {
  if (request.hasHistory) return request.lastPurchaseCategory;
  return 'electronics'; // overall trending across the whole site
}

// recsV2 (the new model, in SHADOW -- the trainee pilot): same criteria for
// buyers with history, but changes the cold-start strategy: instead of the
// site's overall trend, it uses the buyer's REGION trend.
const REGION_TRENDING = { north: 'electronics', south: 'home', east: 'fashion', west: 'sports' };
function recsV2(request) {
  if (request.hasHistory) return request.lastPurchaseCategory;
  return REGION_TRENDING[request.region];
}

// serveWithShadow: the buyer ONLY sees userSees (recs-v1). shadowOnly (recs-v2)
// gets computed in parallel and logged, but never reaches the buyer.
function serveWithShadow(request) {
  const userSees = recsV1(request);
  const shadowOnly = recsV2(request);
  return { requestId: request.requestId, userSees, shadowOnly };
}

const demoRequests = [
  { requestId: 'req-demo-01', hasHistory: true, lastPurchaseCategory: 'sports', region: 'north' },
  { requestId: 'req-demo-02', hasHistory: true, lastPurchaseCategory: 'beauty', region: 'south' },
  { requestId: 'req-demo-03', hasHistory: false, lastPurchaseCategory: null, region: 'north' },
  { requestId: 'req-demo-04', hasHistory: false, lastPurchaseCategory: null, region: 'south' },
];

console.log('=== serveWithShadow on 4 sample requests ===\n');
demoRequests.forEach((r) => {
  const result = serveWithShadow(r);
  console.log(result.requestId + '  userSees=' + result.userSees + '  shadowOnly(recs-v2)=' + result.shadowOnly +
    (result.userSees === result.shadowOnly ? '  [match]' : '  [differ]'));
});

What to expect. Running the file with Node, the output is exactly this:

=== serveWithShadow on 4 sample requests ===

req-demo-01  userSees=sports  shadowOnly(recs-v2)=sports  [match]
req-demo-02  userSees=beauty  shadowOnly(recs-v2)=beauty  [match]
req-demo-03  userSees=electronics  shadowOnly(recs-v2)=electronics  [match]
req-demo-04  userSees=electronics  shadowOnly(recs-v2)=home  [differ]

Notice the pattern: for req-demo-01 and req-demo-02 — buyers with purchase history — the two models always agree, because they share exactly the same criteria for that case. For req-demo-03 — a new buyer in the north region — they also agree, because the site's overall trend (electronics) happens to be the same as that region's trend. But req-demo-04 — a new buyer in the south region — reveals the first real difference: recs-v1 recommends electronics (the overall trend), while recs-v2 recommends home (that specific region's trend). That buyer, right now, never sees home — userSees stays electronics, exactly what recs-v1 decided. The difference only gets logged in shadowOnly, available for review, with no effect on the real shopping experience.

The two properties that make this genuinely "risk-free"

Shadow mode only keeps its promise if two properties hold, and it's worth naming them precisely.

The first is that the shadow never touches the response the user receives. In this lesson's code, that's literal: userSees comes only from recsV1(), and no result from recsV2() can accidentally leak into that variable. In a real system, this property is typically implemented at the infrastructure level — the shadow model runs on a separate endpoint, or receives an async copy of the traffic, so that an error or slowness in recs-v2 can't even delay the response the buyer is waiting for from recs-v1. If the shadow model can, in any way, affect the latency or the content of what the user sees, it's no longer shadow mode — it's, in practice, an experiment with real users, even if nobody called it that.

The second is that the shadow receives the same real traffic as the production model, not a separate test set. This is what distinguishes shadow mode from a simple offline evaluation: recs-v2, in this example, sees exactly the same four requests — same buyer, same moment, same context data — that recs-v1 saw. That's the only way to confirm how the new model would behave with the real, sometimes messy variety of production traffic, instead of only with the clean cases of a test set built in advance.

Common mistakes

Running the new model against test data, and calling it "shadow mode." What happens: the team evaluates recs-v2 against a historical dataset saved for testing, gets good results, and reports that "it's already validated in shadow" — without the model having seen a single real, live Mercado traffic request. Why it happens: running a model against test data is simpler to organize than running it in parallel on production traffic, and the two exercises feel similar because both "don't affect the user." How to spot it: if nobody can point to a real, recent request from a Mercado buyer that recs-v2 processed, the validation was offline, not shadow. How to fix it: as in this lesson's example, shadow mode means the new model receives a copy of the same real traffic the production model also receives — not a sample saved in advance.

Letting the shadow model delay or affect the real response, "just a little." What happens: someone implements the shadow so the application waits for recs-v2's response before continuing, even though it later discards that result — and if recs-v2 is slower than recs-v1, the buyer notices a delay, even without seeing any difference in the content. Why it happens: the simplest way to code "call both models" is sequentially and synchronously, without thinking about how the second model's latency can leak into the user's experience, even if its content doesn't. How to spot it: if the user-perceived latency rises when the shadow is active, compared to when it's off, the shadow is already affecting the real experience, even if the response's content is identical. How to fix it: the shadow should run asynchronously or in genuine parallel — never blocking the response to the user —, exactly the guarantee described in AWS SageMaker's shadow testing documentation in this lesson's resources.

Leaving the shadow running indefinitely, with no plan for what to do with what gets logged. What happens: the team turns on recs-v2's shadow, leaves it running for months, accumulating logs, but never defines when there's enough evidence to make a decision or who's responsible for reviewing it. Why it happens: turning on the shadow feels like the important, hard step; defining the criteria for "when do we already know enough" feels like an administrative detail that can be postponed. How to spot it: if nobody can say, today, how many requests recs-v2's shadow has accumulated or when the result is going to be reviewed, the shadow is a data source with no destination. How to fix it: shadow mode exists to feed a concrete decision — the one shadowCompare() builds in lesson 4 — not as a permanent state. Define, in advance, a reasonable volume or time for shadow, and what happens with the result.

Exercises

Exercise 1 — A fifth request. A new buyer (no history) browses from the east region. Using this lesson's recsV1() and recsV2(), what does the buyer see (userSees), and what only gets logged in shadow (shadowOnly)? Do they match or differ?

See solution

recsV1(), for a buyer with no history, always returns 'electronics' (the overall trend), regardless of region — so userSees = 'electronics'. recsV2(), for the 'east' region, looks up REGION_TRENDING.east, which is 'fashion' — so shadowOnly = 'fashion'. The two differ: the buyer sees electronics, while in shadow, recs-v2 would have recommended fashion. This difference gets logged, with no effect on what the buyer actually saw.

Exercise 2 — Explain it without using the word "shadow." In two or three sentences, explain to someone new on the team what serveWithShadow() does, without using the word "shadow." You can use the trainee pilot analogy.

See solution

One example answer: "It's like having two people solve the same case at the same time, but only one of them has permission to communicate their answer to the customer. The second person also solves the case, with the same information, and their answer gets saved for comparison later — but the customer never sees it or gets affected by it." The central idea, without the technical vocabulary: two decisions computed in parallel on the same real case, but only one with permission to reach the real person.

Exercise 3 — Find the risk. A teammate proposes a variant of serveWithShadow() that first calls recsV2() and, if its result "looks reasonable" under a simple rule, uses it as userSees instead of recsV1(). Why would this no longer be shadow mode, even if the teammate insists on calling it that?

See solution

As soon as recsV2()'s result has any possibility of becoming what the buyer actually sees — even under a condition — recs-v2 stopped being in shadow: it's making real decisions for real users, without anyone having measured yet how differently it decides from recs-v1 or how often. This is, in practice, an experiment with real traffic disguised as shadow mode. The property that defines shadow mode — and that this variant breaks — is that the shadow model never, under any condition, determines what the user receives; that entire decision always belongs to the production model, until an explicit canary — lesson 4 onward — decides otherwise, with evidence and in a controlled way.

Summary and next step

In this lesson you built recsV1(), recsV2(), and serveWithShadow(): the production model keeps deciding everything the buyer sees, while the new model runs in parallel, on the same real traffic, with no effect on that decision. You confirmed, with four sample requests, that the two models agree in most cases — buyers with history, and cold-starts from a region where the local trend matches the overall one — but already start differing in at least one: a new buyer from a region whose trend differs from the overall one.

Before moving on you should be able to: explain the two properties that make a shadow genuinely "risk-free" (it never affects the real response, and it receives real traffic); distinguish shadow mode from an offline evaluation against test data; and anticipate why you need far more than four requests to trust a conclusion about how differently recs-v2 decides.

Lesson 4 takes these exact same two functions, without changing a line, and runs them on a real volume of shadow traffic — building shadowCompare(), the function that turns these individual comparisons into a measurable agreement rate, and into a clear map of where, specifically, the two models differ.

Resources

• Amazon SageMaker AI, "Shadow tests" — docs.aws.amazon.com/sagemaker/latest/dg/shadow-tests.html. Describes this lesson's central guarantee on a real platform: "only the production variant's responses are returned to the calling application" — shadow mode implemented at the infrastructure level, with no risk of the new model affecting the user. In English.
• Microsoft Learn, "Safe rollout for online endpoints" (Azure Machine Learning) — learn.microsoft.com/en-us/azure/machine-learning/how-to-safely-rollout-online-endpoints. Documents the pattern of deploying a new model version alongside the production one before directing real traffic to it, the same idea behind serveWithShadow() in this lesson. In English.