Module 4: Branch by Abstraction

Parallel-run to prove the swap is safe

Next →

Overview

In lesson 4 you set up the flag and verified that switching the implementation doesn't break the callers —but with a comfortable trap: ModernShipping matched the legacy in all the cases, so callers OK? gave True for free—. In reality, the new implementation almost never matches the old one on the first try. A legacy monolith is full of tacit rules —edges nobody documented, exceptions added on a Tuesday five years ago— and the reimplementation, however good, skips one. The central question of this step is: how do you know you can trust the new one before giving it a single real user? The answer is the parallel-run.

A parallel-run is simple and powerful: for each input, you call the two implementations —the old and the new—, compare their results, and report the discrepancies. But with a golden rule: you return to the caller always the legacy's value, never the modern's. The modern runs "in the shadow": it's executed and compared, but its result doesn't reach the user. That way you can validate the new implementation against the old one over real traffic (or over a batch of known cases), catching every difference, without a single one of those differences affecting anyone. The parallel-run is the plane's new engine turned on on the ground, with its readings compared against the old one, before trusting it with a flight with passengers.

This lesson executes it with a real bug. We set up a ModernShippingV1 that forgot a tacit rule of the legacy —that free shipping doesn't apply to international— and we run the parallel-run: the run catches the discrepancy in the international case, in a known order, with the exact number of the difference. We fix it (ModernShippingV2), we run again, and the discrepancy disappears: 0 differences. Only then is it safe to raise the flag. The parallel-run turns "I think the new one is equivalent" into "I measured that the new one is equivalent in these cases," which is a very different assertion.

Connection with the module. Lesson 4 gave you the flag; this one gives you what has to be verified before raising it. The real order of the pattern is: clean parallel-run first (this lesson), flag rollout after (lesson 4). Lesson 6 deletes the legacy when the flag reached 100% and the parallel-run stays clean. Notice the boundary: the parallel-run here is fast and behavioral —it compares the output of two implementations of a function, in memory, over known cases—. The in-depth data parallel-run (reading from two stores during a data migration, with reconciliation in production at scale) is module 6. Here it's a validation tool for the swap; there it's a phase of the data migration. The idea is the same —run the two and compare before trusting—; the object being compared and the scale are different.

An analogy: the tester who bakes the two recipes and compares

Imagine a bakery that wants to replace its old bread recipe —grandma's, handwritten, with margin notes nobody quite understands— with a new one, cleaner and easier to scale. Before selling the new bread to customers, the master baker does something prudent: for a week, they bake the two recipes in parallel, the old and the new, with the same ingredients and the same oven. And they compare the loaves: same weight? same crust? same crumb? same flavor?

What they sell to customers during that week is always the old bread —the tested one, the one people already know—. The new bread is baked, compared, and taken to the internal tasting table; it doesn't reach the counter. If one day the new bread comes out flatter, or with a pale crust, the baker detects it in the tasting —not in a customer's complaint—. They discover the new recipe skipped a rest that grandma's margin note asked for "when it's hot." They adjust the new recipe, bake in parallel again, and compare once more. Only when the new bread comes out identical to the old one, day after day, do they start selling it.

Baking the two recipes and comparing is the parallel-run. Always selling the old bread during the test is the golden rule —the modern runs in the shadow, the user receives the legacy—. The margin note the new recipe skipped is the tacit rule of the legacy the reimplementation forgot. And the internal tasting that catches the flat bread before it reaches a customer is the discrepancy the parallel-run reports before it reaches a user. The bakery doesn't guess whether the new recipe is good: it measures it, in parallel, with the old bread as a reference, without risking the clientele.

Worked example: the parallel-run catches a discrepancy and the validation clears

We're going to set up the parallel-run and use it to catch a parity bug. We have the usual LegacyShipping. And we set up a ModernShippingV1 that has a plausible bug: when reimplementing the free-shipping rule, it wrote if order["order_total"] >= 50.0: cost = 0.0 —forgetting the and order["zone"] != "international" of the legacy—. In isolation, that code looks reasonable; only the parallel-run, comparing against the legacy, gives it away. The parallel-run calls the two, compares, reports discrepancies, and always returns the legacy. We run with V1 (catches the discrepancy), fix to ModernShippingV2, and run again (0 discrepancies).

from typing import Protocol

class ShippingCalculator(Protocol):
    def cost(self, order: dict) -> float: ...

class LegacyShipping:
    def cost(self, order: dict) -> float:
        base = {"local": 5.0, "national": 10.0, "international": 25.0}[order["zone"]]
        cost = base + max(0.0, order["weight_kg"] - 1.0) * 2.0
        if order["order_total"] >= 50.0 and order["zone"] != "international":
            cost = 0.0
        return round(cost, 2)

# --- modern v1: has a BUG. It forgot the tacit rule "free shipping does NOT apply to
# international". In isolation it looks reasonable; only the parallel-run gives it away. ---
class ModernShippingV1:
    def cost(self, order: dict) -> float:
        base = {"local": 5.0, "national": 10.0, "international": 25.0}[order["zone"]]
        cost = base + max(0.0, order["weight_kg"] - 1.0) * 2.0
        if order["order_total"] >= 50.0:            # BUG: missing 'and zone != international'
            cost = 0.0
        return round(cost, 2)

# --- modern v2: fixed, with the tacit rule now replicated. ---
class ModernShippingV2:
    def cost(self, order: dict) -> float:
        base = {"local": 5.0, "national": 10.0, "international": 25.0}[order["zone"]]
        cost = base + max(0.0, order["weight_kg"] - 1.0) * 2.0
        if order["order_total"] >= 50.0 and order["zone"] != "international":
            cost = 0.0
        return round(cost, 2)

ORDERS = [
    {"id": 1, "zone": "local",         "weight_kg": 0.5, "order_total": 20.0},
    {"id": 2, "zone": "national",      "weight_kg": 1.0, "order_total": 30.0},
    {"id": 3, "zone": "international", "weight_kg": 3.0, "order_total": 80.0},
    {"id": 4, "zone": "local",         "weight_kg": 1.0, "order_total": 60.0},
    {"id": 5, "zone": "national",      "weight_kg": 4.0, "order_total": 55.0},
]

# --- parallel-run: calls BOTH, compares, reports the discrepancy, but RETURNS
# the legacy's value (the user never sees the new one's unverified result). ---
def parallel_run(orders, legacy: ShippingCalculator, modern: ShippingCalculator):
    mismatches = []
    for o in orders:
        lc, mc = legacy.cost(o), modern.cost(o)
        if lc != mc:
            mismatches.append((o["id"], o["zone"], lc, mc))
        # the caller ALWAYS receives the legacy's value: parallel-run doesn't expose the new one.
    return mismatches

legacy = LegacyShipping()

print("=== parallel-run with modern v1 (with the bug) ===")
mm = parallel_run(ORDERS, legacy, ModernShippingV1())
print(f"{'order':>6}{'zone':>16}{'legacy':>9}{'modern':>9}   status")
print("-" * 49)
for o in ORDERS:
    lc, mc = legacy.cost(o), ModernShippingV1().cost(o)
    flag = "OK" if lc == mc else "DISCREPANCY"
    print(f"{o['id']:>6}{o['zone']:>16}{lc:>9}{mc:>9}   {flag}")
print(f"\nDiscrepancies: {len(mm)}  ->  {mm}")
print("Do NOT switch the flag: the new one differs from the legacy in a known case.\n")

print("=== parallel-run with modern v2 (fixed) ===")
mm2 = parallel_run(ORDERS, legacy, ModernShippingV2())
print(f"{'order':>6}{'zone':>16}{'legacy':>9}{'modern':>9}   status")
print("-" * 49)
for o in ORDERS:
    lc, mc = legacy.cost(o), ModernShippingV2().cost(o)
    flag = "OK" if lc == mc else "DISCREPANCY"
    print(f"{o['id']:>6}{o['zone']:>16}{lc:>9}{mc:>9}   {flag}")
print(f"\nDiscrepancies: {len(mm2)}  ->  {mm2}")
print("0 discrepancies in the known cases: NOW it's safe to raise the flag.")

What to expect. When you run the file, the output is exactly this:

=== parallel-run with modern v1 (with the bug) ===
 order            zone   legacy   modern   status
-------------------------------------------------
     1           local      5.0      5.0   OK
     2        national     10.0     10.0   OK
     3   international     29.0      0.0   DISCREPANCY
     4           local      0.0      0.0   OK
     5        national      0.0      0.0   OK

Discrepancies: 1  ->  [(3, 'international', 29.0, 0.0)]
Do NOT switch the flag: the new one differs from the legacy in a known case.

=== parallel-run with modern v2 (fixed) ===
 order            zone   legacy   modern   status
-------------------------------------------------
     1           local      5.0      5.0   OK
     2        national     10.0     10.0   OK
     3   international     29.0     29.0   OK
     4           local      0.0      0.0   OK
     5        national      0.0      0.0   OK

Discrepancies: 0  ->  []
0 discrepancies in the known cases: NOW it's safe to raise the flag.

Read the first run, the one with ModernShippingV1. Four rows give OK and one gives DISCREPANCY: order 3, international with total 80.0. The legacy charges 29.0 (base 25.0 + surcharge of 4.0 for the 2 kg extra), but ModernShippingV1 charges 0.0. Why? Because V1 forgot the tacit rule: when the order exceeds 50.0, V1 makes it free, without excluding international as the legacy does. The summary line says it exactly: Discrepancies: 1 -> [(3, 'international', 29.0, 0.0)]. That (3, 'international', 29.0, 0.0) is an actionable report: in order 3, zone international, the legacy gives 29.0 and the modern gives 0.0. It's not "something's wrong"; it's "here, exactly, they differ, and by how much."

And notice what did not happen: that order 3 never received 0.0 as a response. The parallel-run always returns the legacy's value (29.0); the modern's 0.0 was computed, compared, and reported —but didn't reach any user—. If this were real production, the customer of order 3 would have paid the correct 29.0, and the team would have found out about the modern's bug from the discrepancy report, not from an angry customer who received improper free shipping. That's the safety of the parallel-run: it validates the new one over real cases without its errors reaching anyone.

The second run is with ModernShippingV2, where the missing and order["zone"] != "international" was added. Now the five rows give OK, including order 3 (29.0 in both), and the summary says Discrepancies: 0 -> []. That empty list is the green light: the new implementation matches the old one in all the known cases. 0 discrepancies in the known cases: NOW it's safe to raise the flag. The parallel-run turned a hunch ("v1 looked fine") into a measurement (v1 fails on international; v2 doesn't), and only after the clean measurement is the flag rollout of lesson 4 authorized.

Deep dive: the golden rule, and what is 'safe' and what isn't

The structure of the parallel-run fits in a diagram:

                        parallel_run(order)
                              │
             ┌────────────────┴────────────────┐
             ▼                                  ▼
      legacy.cost(order)                 modern.cost(order)
             │                                  │
             │  (RETURNED to the caller)         │  (computed, NOT returned)
             ▼                                  ▼
        response to the user   ◄── compare ──►  only for the report
                                     │
                          if they differ: log discrepancy

The golden rule —always return the legacy— is what makes the parallel-run safe. As long as the new implementation isn't validated, its result must not reach a user, because it could be wrong (like V1's 0.0). The parallel-run runs it "in the shadow": it executes it to be able to compare it, but discards its output toward the user. This has a cost —you compute twice— but it buys something very valuable: validation over real traffic without exposure. When the parallel-run has been at 0 discrepancies for enough time (or enough cases), there you start returning the modern —which is, precisely, raising the flag of lesson 4—.

It's worth being precise about what a parallel-run demonstrates and what it doesn't. It demonstrates parity in the cases you ran: if you compared 5 cases and they gave 0 discrepancies, you know they match in those 5. It doesn't demonstrate universal parity: there could be a sixth case, not covered, where they differ. That's why, in reality, the parallel-run is run over many cases —ideally over a sample of the real production traffic, for days— so that "known cases" gets close to "all the cases that really happen." The more real cases it covers without discrepancies, the more justified the confidence. Five cases chosen by hand, as in the example, illustrate the mechanics; a serious validation runs thousands.

And an important decision the parallel-run forces: when a discrepancy appears, is the legacy right or the modern? Almost always the legacy —it's the current behavior, the one the business already lives, and the pattern's rule is migrate to parity first—. V1's discrepancy (international free) is a clear case: the legacy is right, international must not be free, and the modern is fixed to match. But occasionally the discrepancy reveals a bug in the legacy that the modern "fixed" by accident. There the rule is not to decide it hot: first you take the modern to exact parity with the legacy (bug included), complete the migration, and then, in a separate and deliberate change, you fix the bug —so as not to mix "I migrated" with "I changed behavior"—. The parallel-run doesn't decide who's right; it shows you where they differ so you decide, and the default decision is parity.

Common mistakes

Returning the modern's result during the parallel-run. What happens: the team, in a hurry to "use" the new implementation already, makes the parallel-run return the modern's value and only log if it differs from the legacy. Why it happens: it seems like a shortcut —"if we're already computing it, why not use it?"—. How to spot it: a user receives a result from the unvalidated implementation; V1's discrepancy (0.0 on international) would have reached a customer. How to fix it: the golden rule is non-negotiable —during the parallel-run you always return the legacy—. The modern runs in the shadow, is compared, is reported, but isn't exposed. Returning the modern is raising the flag, and that's only done after sustained 0 discrepancies. A parallel-run that returns the modern isn't a parallel-run: it's a rollout without validation, under the wrong name.

Concluding "it's equivalent" with too few cases. What happens: the parallel-run gives 0 discrepancies over five cases and the team declares universal parity and raises the flag to 100%. Why it happens: five OK in a row give a certainty the sample doesn't support. How to spot it: your validation covers a handful of cases chosen by hand, not the variety of the real traffic; an order with a weird combination (zone + weight + total) that isn't in the sample could differ. How to fix it: run the parallel-run over many cases —ideally a sample of the production traffic, for days— so that "known cases" resembles "all the real cases." And even so, raise the flag gradually (lesson 4): the 10% rollout with the old one as a net is the second layer of safety in case the parallel-run didn't cover some case. Parity measured over many cases + gradual rollout, not five cases + a jump to 100%.

"Fixing" a legacy bug inside the reimplementation, without deciding it. What happens: the parallel-run shows a discrepancy, the team looks and thinks "the legacy is wrong here, the modern does it right," and lets the modern differ —mixing the migration with a behavior correction—. Why it happens: some of the legacy's quirks are bugs, and fixing them "along the way" feels efficient. How to spot it: at the end of the migration, the system doesn't behave the same as before in some cases, and nobody decided that explicitly —it snuck into the "I migrated"—. How to fix it: separate the two things. First migrate to exact parity with the legacy, bug included, so that the migration is purely structural (same output, better code). Then, in a separate and deliberate change, fix the bug with its own validation and its own record. That way, if something breaks, you know whether it was the migration or the correction —not an ambiguous mix of the two—. The parallel-run shows you the discrepancy; the discipline is not to resolve it by secretly changing behavior.

Exercises

Exercise 1 — The bakery that bakes in parallel. In the analogy, the baker bakes the old recipe and the new one in parallel for a week, comparing them, but always sells the old bread. (a) What corresponds, in the code's parallel-run, to "always sell the old bread"? (b) What is the "grandma's margin note" the new recipe skipped? (c) Why is comparing the loaves in an internal tasting better than finding out from a customer's complaint?

See solution

(a) It corresponds to the golden rule: the parallel-run always returns the legacy's value to the caller. The modern is computed and compared (it's baked and tasted), but its result doesn't reach the user (it isn't sold). Always sell the old bread = always return the legacy while the new one isn't validated.

(b) It's a tacit rule of the legacy the reimplementation forgot: in the example, "free shipping doesn't apply to international." Like the margin note ("an extra rest when it's hot"), it's a non-obvious detail of the old behavior the new, cleaner version skipped without noticing. The parallel-run gives it away by comparing against the legacy.

(c) Because the internal tasting catches the defect before it reaches a customer: the baker tastes the new bread themselves, in the shadow, and if it comes out flat they detect it without any customer receiving a bad loaf. In the code, the parallel-run catches the discrepancy (international charging 0.0) in the report, not in a customer who received improper free shipping. Finding out from a customer's complaint means the error already caused damage; finding out from the shadow comparison means you caught it at no cost. The passive validation —run and compare without exposing— is what buys that safety.

Exercise 2 — Read the discrepancy. The parallel-run with V1 reported Discrepancies: 1 -> [(3, 'international', 29.0, 0.0)]. (a) Interpret that tuple field by field. (b) What rule of the legacy did ModernShippingV1 forget, and how do you know from the number? (c) During this run, what value did the "user" of order 3 receive, and why does that matter?

See solution

(a) The tuple (3, 'international', 29.0, 0.0) says: in order 3, of zone international, the legacy computed 29.0 and the modern (V1) computed 0.0. It's an actionable report: it identifies the exact case (order 3, international), and the two figures that differ (29.0 vs 0.0), so you can go straight to the rule that's failing.

(b) It forgot that free shipping doesn't apply to international. You know from the number: order 3 has total 80.0, which exceeds the free-shipping threshold (50.0). The legacy does not make it free because it's international (it charges the normal 29.0: base 25.0 + 4.0 surcharge). V1 does make it free (0.0) because its condition is only total >= 50.0, without excluding international. The difference from 29.0 to 0.0 is exactly the shipping cost V1 gave away by skipping that exclusion.

(c) It received 29.0 —the legacy's value—, not the modern's 0.0. It matters because it demonstrates the golden rule in action: even though V1 had the bug and computed 0.0, that value never reached the user; the parallel-run returned the legacy and only reported the difference. The customer paid the correct amount and the team found out about the bug from the report, not from an incident. If the parallel-run had returned the modern, that customer would have received improper free shipping —the bug would have reached production—.

Exercise 3 — Legacy or modern right? A team's parallel-run reports a discrepancy: for a certain order, the legacy charges 12.0 and the modern charges 10.0. On investigating, they discover the legacy adds a "handling surcharge" of 2.0 added years ago that the business today considers an error that should be removed. (a) According to the pattern's rule, which implementation must win during the migration, and why? (b) What would they do with the surcharge the business wants to remove? (c) Why is it important not to mix the two things?

See solution

(a) During the migration the legacy must win (charge 12.0): the pattern's rule is migrate to parity first. The goal of this phase is for the new implementation to behave the same as the old one —including the 2.0 surcharge, even if it looks like an error— so that the migration is purely structural: same observable behavior, better code. ModernShipping must be fixed to include the surcharge and match the legacy at 12.0.

(b) The surcharge the business wants to remove is eliminated in a separate and deliberate change, after the migration is complete and stable: its own decision, its own validation, its own record (ideally an ADR, which is the topic of the decisions guide). It isn't resolved by letting the modern differ "along the way" during the migration.

(c) Because mixing them makes it impossible to reason about what caused what. If the modern differs from the legacy and that includes both the migration and the removal of the surcharge, when something breaks you won't know whether it was the change of implementation or the change of behavior. By separating them —migrate to exact parity first, change behavior after— each step has a single cause and a single validation. The migration proves "same output, better code"; the surcharge change proves "new business rule, approved." Two clean assertions instead of one ambiguous one. Besides, reverting is simpler: you can undo the surcharge change without touching the migration, or vice versa.

Summary and next step

In this lesson you did the validation that authorizes the rollout: the parallel-run to prove the swap is safe. You saw, with the bakery that bakes the two recipes in parallel and always sells the old bread, that you can validate the new implementation against the old one without exposing its errors to anyone. And you executed it with a real bug: a ModernShippingV1 that forgot the tacit rule "international isn't free," which the parallel-run caught in order 3 with the exact number of the difference (29.0 vs 0.0) —without that 0.0 reaching any user—. You fixed it to ModernShippingV2, ran again, and got Discrepancies: 0: the green light to raise the flag. You learned the golden rule (always return the legacy during the test), what a parallel-run demonstrates (parity in the cases run, not universal), and why the discrepancies are resolved by default in favor of the legacy, leaving the behavior corrections for a separate change.

Before moving on you should be able to: explain what a parallel-run is and its golden rule; read a discrepancy report and translate it to the failing rule; say why five clean cases don't authorize a jump to 100%; and distinguish "migrate to parity" from "fix a legacy bug," and why they aren't mixed.

Lesson 6 does the last step, the one almost nobody does: deleting the old implementation. When the flag reached 100% and the parallel-run stays clean, LegacyShipping no longer receives traffic or contributes anything —it just weighs—. You're going to run the inventory of moving parts before and after the deletion, and measure the cost of leaving the two "just in case": a new requirement that forces editing each live implementation, and the real drift (a KeyError in the run) when one is forgotten. Leaving the two forever is the eternal migration, the pattern's silent failure; deleting the old one is what pays for the migration.

Resources

  • Martin Fowler, "Patterns of Legacy Displacement" — martinfowler.com/articles/patterns-legacy-displacement. Run the two implementations over the same input and compare their results, with the old one as the source of truth, to validate the new one without exposing it; the parallel run as a named pattern is from Sam Newman (Monolith to Microservices). The direct reading of this lesson. In English.
  • Michael Feathers, Working Effectively with Legacy Code (Prentice Hall, 2004) — the characterization tests as the way to capture the legacy's behavior (including its tacit quirks) that the parallel-run then verifies the modern reproduces. The connection with module 2 of this guide. In English.
  • GitHub, "Scientist" — github.com/github/scientist. The library that popularized the parallel-run in production ("science experiments"): it runs the old and new code in parallel, returns the old, and reports the discrepancies. The real implementation of what this lesson simulates. In English.
  • Sam Newman, Monolith to Microservices (O'Reilly, 2019), ch. 3, section "Parallel Run" — the parallel-run as a verification technique during a monolith's migration, with the warning to compare it over sufficient real traffic. In English.
← PreviousThe feature flag as the switch pointNext →Deleting the old implementation