Dwell time: how long to wait before trusting a stage

Description

The previous lessons gave each ramp stage a correct size (lesson 3), a correct advance criterion (lesson 4), and a well-chosen population (lesson 6). One piece is missing: time. A guardrail that "looks good" barely five minutes after turning on a stage isn't a trustworthy measurement yet — it's a photo taken too soon. This lesson introduces dwell time: the minimum time a stage needs to stay active before its data is good enough to decide anything.

Connection to the module. This lesson adds a second condition to the advance criterion lesson 4 formalized: it's not enough for the guardrail to be within the ceiling — you also need to have waited long enough, and accumulated enough volume, before trusting that measurement. It's the ramp's last piece before lesson 8's mini-project, which is going to bring the module's six pieces together over recommendations' full rollout.

An analogy: tasting the soup before it's done boiling

Nobody tastes whether a soup came out well salted thirty seconds after adding the salt — the flavor hasn't distributed through the liquid yet, and a spoonful at that moment can come out bland or, conversely, straight-up salty, depending on how close it landed to exactly where the salt fell. You need to wait for it to boil for a while, for everything to mix, before tasting a spoonful says anything trustworthy about the whole pot's flavor. Tasting too soon isn't just unhelpful — it can lead to an actively wrong conclusion, in either direction.

A rollout stage's guardrail has the same problem. Five minutes after raising the canary to 1%, maybe only a handful of users have gone through the new code — the p95Latency measurement at that point could be dominated by a single slow request, or by pure chance be free of any slow request, with neither case saying anything trustworthy about how that stage is going to behave once it's truly "boiled" long enough.

Worked example: stageDwellCheck() — time and volume, both at once

A dwell time check needs two conditions at the same time: enough time elapsed, and enough accumulated data volume. Neither one, alone, is enough:

// stageDwellCheck: before advancing, TWO things are needed -- minimum elapsed
// time (minDwellHours) AND minimum data volume (minSampleSize). A guardrail
// that "looks good" with very little data, or very little time, isn't yet a
// trustworthy signal -- it's lucky noise.
function stageDwellCheck({ stage, elapsedHours, minDwellHours, exposedUsers, minSampleSize }) {
  const enoughTime = elapsedHours >= minDwellHours;
  const enoughSample = exposedUsers >= minSampleSize;
  const ready = enoughTime && enoughSample;
  return { stage, elapsedHours, minDwellHours, enoughTime, exposedUsers, minSampleSize, enoughSample, ready };
}

const checks = [
  { stage: 'canary 1%', elapsedHours: 6, minDwellHours: 4, exposedUsers: 2500, minSampleSize: 1000 },
  { stage: 'rollout 10%', elapsedHours: 1, minDwellHours: 12, exposedUsers: 25000, minSampleSize: 5000 },
  { stage: 'rollout 50%', elapsedHours: 18, minDwellHours: 24, exposedUsers: 125000, minSampleSize: 20000 },
];

console.log('=== stageDwellCheck: minimum time AND sample before advancing ===\n');
checks.forEach((c) => {
  const r = stageDwellCheck(c);
  console.log(r.stage.padEnd(14) + 'elapsed=' + String(r.elapsedHours).padStart(2) + 'h/' + String(r.minDwellHours).padStart(2) + 'h' +
    '  sample=' + r.exposedUsers.toLocaleString('en-US').padStart(7) + '/' + r.minSampleSize.toLocaleString('en-US').padStart(6) +
    '  ready=' + r.ready);
});

const notReady = checks.map(stageDwellCheck).filter((r) => !r.ready);
console.log('\nStages that are NOT ready to advance yet: ' + notReady.map((r) => r.stage).join(', ') + '.');

What to expect. Running the file with Node, the output is exactly this:

=== stageDwellCheck: minimum time AND sample before advancing ===

canary 1%     elapsed= 6h/ 4h  sample=  2,500/ 1,000  ready=true
rollout 10%   elapsed= 1h/12h  sample= 25,000/ 5,000  ready=false
rollout 50%   elapsed=18h/24h  sample=125,000/20,000  ready=false

Stages that are NOT ready to advance yet: rollout 10%, rollout 50%.

Look closely at the rollout 10% row: it has 25,000 exposed users, five times the required minimum sample (5,000) — data volume is more than covered. And yet, ready: false, because only 1 of the required 12 hours has passed. This row is exactly the case that justifies why the model demands both conditions at once: if stageDwellCheck() only looked at volume, this stage would have given ready: true with barely one hour of data — enough time for, say, a single hour's traffic spike (lunch hour, a one-off event) to completely distort the p95Latency measurement, with that bias not yet visible.

Why volume alone isn't enough — and why time alone isn't either

Today's example reveals something worth naming explicitly: volume and time capture different risks, and neither covers the other's gap. Volume (exposedUsers >= minSampleSize) protects against statistical noise — too few events, and any result is more luck than signal, exactly the problem you saw in lesson 3 with a too-small canary. Time (elapsedHours >= minDwellHours) protects against a different risk: variation by time of day or weekly cycle. A system can behave completely differently on a Monday at 9am (high traffic, weekend jobs just wrapping up) than on a Saturday at 3am (minimal traffic) — and one hour of data, no matter how many users are in that hour, can never capture that variation.

rollout 50% in the example shows the opposite case: it nearly meets the time requirement (18 of 24 hours) and has plenty of volume (125,000 against a minimum of 20,000) — but ready is still false, because that last time condition is missing. There are no shortcuts: both conditions have to be met at the same time, one can't compensate for the other.

Common mistakes

Zero dwell time: advancing before the data arrives, right as the stage turns on. What happens: someone checks the guardrail a few minutes after raising a new stage, sees it within the ceiling, and advances to the next one immediately. Why it happens: the anxiety to move fast — especially if previous stages already passed with no problem — makes a first "clean" look feel like sufficient confirmation. How to spot it: as in the example's rollout 10% row, elapsed time is a small fraction of the required minimum, even though the guardrail looks good at that instant. How to fix it: set an explicit minDwellHours for each stage, before turning it on — don't decide it on the fly by looking at how good the number looks at that moment — and stick to it even if the guardrail looks perfect from the first minute.

Using the same fixed dwell time for every stage, without adjusting for its expected traffic volume. What happens: the team defines "we wait 4 hours at each stage" as a general rule, without distinguishing that the 1% canary takes much longer to accumulate the same data volume than the 50% does. Why it happens: a fixed, simple rule is easier to remember and communicate than a different calculation for each stage. How to spot it: compare how long it takes each stage to reach its minSampleSize — if a stage with much less traffic (the canary) needs much more time than a stage with more traffic (the 50%) to gather the same volume, a fixed time rule is over-waiting on one stage and under-waiting on another. How to fix it: as stageDwellCheck() does, define the minimum dwell time based on how much traffic you need to accumulate at that specific stage, not as a round number applied equally to all of them.

Confusing "the guardrail passed once, at an instant" with "the guardrail is stable." What happens: the dashboard gets looked at just once, at a single moment, p95Latency shows under the ceiling, and the stage gets concluded to have "passed" — without having observed behavior across the whole dwell time. Why it happens: a single clean snapshot feels like sufficient evidence, especially if it confirms what the team already expected to see. How to spot it: nobody can describe how the guardrail behaved throughout the full dwell time — they can only cite the last number they saw. How to fix it: dwell time isn't just "wait a while and look once at the end" — it's accumulating enough measurements throughout that period to confirm the guardrail stayed within the ceiling consistently, not just at the instant someone decided to look.

Exercises

Exercise 1 — Calculate whether a stage is ready. The team reviews rollout 10% again, six hours after the example's check: now elapsedHours: 12, with exposedUsers: 30000 (the rest of the values the same: minDwellHours: 12, minSampleSize: 5000). Is it ready to advance? Use stageDwellCheck() mentally.

See solution

enoughTime: 12 >= 12 → true. enoughSample: 30000 >= 5000 → true. ready: true. The stage is now ready — it meets exactly the minimum time (not a minute more) and has more than enough sample volume. It's worth noting "exactly the minimum" is an edge case: in practice, many teams prefer a small extra margin above the strict minimum before advancing, especially if the next stage is one that exposes a big jump of new people (lesson 5).

Exercise 2 — Design a new stage's dwell time. If Mercado adds the intermediate 5% stage (as in lesson 5's exercise 1), and wants its minSampleSize to be proportional to its size relative to rollout 10% (which has minSampleSize: 5000 with 25,000 expected users), what minSampleSize would be reasonable for the 5% stage (which exposes 12,500 users)?

See solution

Keeping the same ratio (minSampleSize / exposedUsers = 5000 / 25000 = 0.2, that is, a minimum sample equivalent to 20% of the stage's expected volume), the 5% stage would need minSampleSize: round(12500 * 0.2) = 2500. It's not the only reasonable way to calculate it — you could also set an equal absolute minimum for every stage, like lesson 3's signal threshold does — but scaling the minimum to each stage's size avoids unnecessarily demanding the same absolute volume from a much smaller stage as from a bigger one.

Exercise 3 — Explain the mistake in your own words. A teammate says: "We have 25,000 users at the 10% stage, that's a ton of people — no need to wait any longer, we already have enough data." Using what you saw in this lesson, explain in 2-3 sentences why that claim, though true about volume, isn't enough to decide to advance.

See solution

The claim is right about volume — 25,000 is, in fact, five times that stage's minimum sample — but volume and time protect against different risks: volume ensures there are enough events for the signal not to be pure statistical noise (lesson 3), while time ensures those events cover a real system's normal variation — different times of day, different days of the week. Twenty-five thousand users gathered in a single peak hour say nothing about how the system behaves overnight, or on a weekend — that's why stageDwellCheck() demands both conditions at once, and having plenty of one doesn't make up for lacking the other.

Summary and next step

In this lesson you added the ramp's last piece: dwell time. With stageDwellCheck() you saw that a stage needs to meet two conditions at once — minimum elapsed time and minimum sample volume — before its guardrail measurement is trustworthy, and that a large volume in little time (like rollout 10%'s 25,000 users in barely 1 hour) doesn't make up for lacking enough time to capture a real system's normal variation.

Before moving on you should be able to: explain why time and volume protect against different risks; and calculate whether a stage is ready to advance given its elapsed time, accumulated volume, and their required minimums.

With this, the ramp's six pieces are complete: canary size (lesson 3), the advance criterion (lesson 4), the growing blast radius (lesson 5), the rings (lesson 6), and dwell time (this lesson) — all built on rolloutPlan()'s structure from lesson 2. Lesson 8, the mini-project, asks you to bring the six pieces together and simulate recommendations' full ramp, start to finish.

Resources

• Martin Fowler (bliki), "CanaryRelease" — martinfowler.com/bliki/CanaryRelease.html. Contrasts a canary's typical duration — minutes or hours — against a statistically significant A/B experiment's — which can take days — a useful nuance for calibrating how long each stage's dwell time should be based on its purpose.
• Google SRE, "Reliable Product Launches at Scale" — sre.google/resources/book-update/reliable-product-launches-at-scale. The Google SRE book chapter on how the organization coordinates reliable launches at scale, including sustained review of a launch over time, not just at the moment of turning it on.