Module 8: Project — Modernizing a Slice of Mercado
Migrate the slice's data
Overview
In the previous lesson you extracted the catalog to its own service and saw that the extraction requires the service to own its data —phase 3, moving the data from the shared DB (shared_db) to its own DB (owned_db)—. The method's fourth step —module 6 turned into action— is to execute that move without downtime: moving the catalog's data from one store to the other while Mercado keeps selling, without losing or corrupting a single row, and without turning off the system for even a second.
The move has a cycle with an order that isn't negotiable. First the dual-write is turned on: from that moment, every write lands in both stores, so the front is covered and nothing new is lost. Then the backfill copies the historical data from the old store to the new one, translating the old model to the clean Product. Then —and this is the piece that separates a serious data migration from a leap of faith— the parallel-run compares the new against the old, row by row, and reports the discrepancies: rows that are missing, fields that are mistranslated. The reconciliation fixes the cause of those discrepancies until it brings them to zero. And only then, with the parallel-run green, is the read-switch done: the reads move to the new store, with the old one hot as backup.
The hard rule of this step, the one you're going to see executed, is a single one: you don't switch a read to the new store while the discrepancies aren't zero. This example's backfill has two real bugs —it skips the inactive products (a row ends up missing) and mistranslates a product's price (a mistranslated field)—, and the parallel-run catches them before the switch, with the reads still coming from the old, so no user sees the gap. You reconcile the cause, the parallel-run returns to zero, and only then do the reads move. The parallel-run is the difference between catching the bug in the migration or discovering it in production through a complaint.
Connection with the module. This is the method's fourth step (M6), and it leans directly on step 3: the return ACL you put in place in lesson 4 is what keeps the monolith's contract intact while the data moves underneath —the monolith keeps receiving its old model even though the data's source changes from shared_db to owned_db—. This step produces the verified new store (discrepancies = 0) that lesson 6 will measure as part of the progress. Notice the boundary: here we execute the idea of the data migration —dual-write, backfill, compare, reconcile, switch— simulated in memory, over five rows. Doing this at real scale in production —CDC, resumable batched backfill, automated reconciliation of billions of rows— belongs to the Data Engineering ecosystem; lesson 8 tells you where to go for that.
An analogy: moving a clinic's records archive without closing
Think of a clinic that's changing its patient records archive —from the paper it's always used to a new digital system— with an inviolable rule: the clinic doesn't close for even a day. Patients keep arriving, notes keep being written, histories keep being updated. There's no maintenance weekend to move everything at once.
This is how a sensible clinic does it. First, it starts writing every new note in both systems (dual-write): from that day, every visit is recorded on paper and in digital, so nothing new is lost. Then it digitizes the historical archive (backfill), without stepping on the fresh notes that already landed in digital. Then it audits: compares records from paper against digital (parallel-run), and finds real problems —the records of discharged patients weren't digitized (someone filtered "active only" by mistake), and a patient had a medication dose copied wrong—. It fixes the cause and re-digitizes (reconciliation): fixes the filter and the copy, and the audit runs again: zero differences. Only then do the doctors start consulting the digital (read-switch), with the paper kept updated for a while just in case.
Mercado's catalog is the clinic's archive; the products are the records. The bug that skips the inactive ones is the filter that skipped the discharged patients; the mistranslated price is the badly copied dose. And the rule is the same: you don't switch to reading from the digital while the audit doesn't come out zero. A doctor who consults the digital before the audit matches up could read a badly copied dose; a catalog read switched before the parallel-run comes out zero would serve a mistranslated price. The audit before the switch is what makes the move safe.
Worked example: the complete cycle with two caught discrepancies
We're going to execute the catalog's data migration from shared_db (old model) to owned_db (Product model). The DataMigration class integrates module 6's pieces. The backfill has two bugs: it skips the inactive products (the webcam-hd, prod 3, ends up missing) and mistranslates prod 4's price (drops a digit). The parallel-run compares row by row and catches them; the reconciliation fixes the cause and re-runs the backfill; the parallel-run returns to zero; and only then is the read-switch done.
# Step 4 of the method: migrate the catalog's data from shared_db (old model) to
# owned_db (Product) WITHOUT downtime. dual_write + backfill fill the new one, the
# parallel_run compares row by row and finds 2 discrepancies, the reconciliation
# fixes them, the parallel_run returns to 0, and ONLY then is the read_switch done.
from dataclasses import dataclass
@dataclass
class Product:
id: int
name: str
price_cents: int
active: bool
# --- Canonicalization: to compare the old model with the new one they're brought to a
# same form. If the new matches the old, canon_old == canon_new. ---
def canon_old(row):
return (row["prod_id"], row["desc"], int(row["prc_cents"]), row["act"] == "Y")
def canon_new(p):
return (p.id, p.name, p.price_cents, p.active)
class DataMigration:
def __init__(self):
# The catalog's old store, in the legacy model (5 products).
self.old_store = {
1: {"prod_id": 1, "desc": "SSD 1TB", "prc_cents": "8999", "act": "Y"},
2: {"prod_id": 2, "desc": "USB-C Hub", "prc_cents": "3499", "act": "Y"},
3: {"prod_id": 3, "desc": "Webcam HD", "prc_cents": "5999", "act": "N"},
4: {"prod_id": 4, "desc": "Mech Kbd", "prc_cents": "7999", "act": "Y"},
5: {"prod_id": 5, "desc": "Mouse Pro", "prc_cents": "2499", "act": "Y"},
}
self.new_store = {} # owned_db, in the Product model
self.write_to_new = False # dual_write OFF at startup
self.read_source = "old"
self.backfill_skips_inactive = True # BUG 1: the backfill skips inactive ones
self.backfill_drops_digit = True # BUG 2: mistranslates the price of prod 4
def to_modern(self, row):
price = int(row["prc_cents"])
if self.backfill_drops_digit and row["prod_id"] == 4:
price = price // 10 # BUG 2: drops a digit (7999 -> 799)
return Product(row["prod_id"], row["desc"], price, row["act"] == "Y")
def backfill(self): # idempotent + insert/fix-if-needed
copied = 0
for pid, row in self.old_store.items():
if self.backfill_skips_inactive and row["act"] == "N":
continue # BUG 1: skips the inactive ones (prod 3)
candidate = self.to_modern(row)
if pid not in self.new_store or self.new_store[pid] != candidate:
self.new_store[pid] = candidate
copied += 1
return copied
def parallel_run(self): # compares row by row, reports the type
discrepancies = []
for pid, row in self.old_store.items():
if pid not in self.new_store:
discrepancies.append((pid, "MISSING_IN_NEW"))
elif canon_old(row) != canon_new(self.new_store[pid]):
discrepancies.append((pid, "FIELD_MISMATCH"))
return discrepancies
m = DataMigration()
print("Catalog data migration: shared_db (old) -> owned_db (Product)\n")
# Step 1: turn on dual_write (the front already lands in both) and run the BUGGY backfill.
m.write_to_new = True
copied = m.backfill()
disc = m.parallel_run()
print(f"1. dual_write ON + backfill (with bugs): copied {copied} rows to new.")
print(f" parallel_run -> {len(disc)} discrepancies:")
for pid, kind in disc:
print(f" prod {pid}: {kind}")
print(" hard rule: discrepancies != 0 -> the read_switch is NOT done yet.\n")
# Step 2: reconcile by fixing the CAUSE (the two bugs) and re-run the backfill.
m.backfill_skips_inactive = False # fix BUG 1
m.backfill_drops_digit = False # fix BUG 2
copied = m.backfill()
disc = m.parallel_run()
print(f"2. reconcile (fix the cause) + re-backfill: fixed/copied {copied} rows.")
print(f" parallel_run -> {len(disc)} discrepancies.\n")
# Step 3: read_switch, allowed ONLY because discrepancies == 0.
if not disc:
m.read_source = "new"
print("3. read_switch: parallel_run at 0 -> reads move to owned_db (new).")
print(f" read_source = {m.read_source}. The old one stays as backup.\n")
print(f" old_store: {len(m.old_store)} rows new_store: {len(m.new_store)} rows "
f"read={m.read_source}")
print(" The catalog's data moved WITHOUT downtime: the parallel_run caught the")
print(" missing row (inactive) and the mistranslated field BEFORE the switch; only with")
print(" discrepancies=0 were the reads switched. You don't switch a read with a gap.")What to expect. When you run the file, the output is exactly this:
Catalog data migration: shared_db (old) -> owned_db (Product)
1. dual_write ON + backfill (with bugs): copied 4 rows to new.
parallel_run -> 2 discrepancies:
prod 3: MISSING_IN_NEW
prod 4: FIELD_MISMATCH
hard rule: discrepancies != 0 -> the read_switch is NOT done yet.
2. reconcile (fix the cause) + re-backfill: fixed/copied 2 rows.
parallel_run -> 0 discrepancies.
3. read_switch: parallel_run at 0 -> reads move to owned_db (new).
read_source = new. The old one stays as backup.
old_store: 5 rows new_store: 5 rows read=new
The catalog's data moved WITHOUT downtime: the parallel_run caught the
missing row (inactive) and the mistranslated field BEFORE the switch; only with
discrepancies=0 were the reads switched. You don't switch a read with a gap.
Read the three steps, because together they are the complete cycle of a verified data migration.
Step 1 — dual-write and the backfill with bugs. The dual-write is turned on (the front already lands in both stores) and the backfill is run, which copied 4 rows to the new —not the old's 5—. The parallel-run compares the old's 5 products against the new and finds 2 discrepancies, each with its type:
prod 3: MISSING_IN_NEW. Thewebcam-hdis inactive (act: "N"), and the backfill skipped it (the skip-inactive bug). It's in the old, absent in the new.prod 4: FIELD_MISMATCH. Themech-kbdwas copied, but with the price mistranslated: the backfill dropped a digit (7999 → 799). It's in both stores, but the price field doesn't match.
And here the hard rule acts: discrepancies != 0 → the read_switch is NOT done. The migration stops. Notice the protection: the two bad rows are still read correctly, because the reads still come from the old (read=old). The gap was detected before it did harm.
Step 2 — reconcile the cause. The two bugs are fixed at their cause (backfill_skips_inactive = False and backfill_drops_digit = False) and the backfill is re-run. Because it's idempotent and fixes by difference, it fixed/copied exactly the 2 rows that were wrong: it inserted the missing webcam-hd and overwrote the mech-kbd with the correct price, without touching the 3 rows that were already right. The parallel-run runs again and now marks 0 discrepancies. Notice that the cause was fixed, not the symptom: fixing the filter covers any inactive one (not just the webcam-hd), and fixing the translation covers any price (not just prod 4's).
Step 3 — read-switch. With the parallel-run at 0, and only because of that, the reads move to owned_db (read_source = new). The old stays hot as backup. The catalog's data now lives in the service's own store, verified row by row, and the system never went down: in the end, old_store: 5 rows, new_store: 5 rows, read=new.
The cycle tells the complete story of a data migration in three steps: a backfill with two real bugs, a parallel-run that caught them before exposing them, a reconciliation that fixed the cause down to zero, and a read-switch that only went through with the discrepancies at zero. The hard rule —not switching a read to the new with an open gap— is what separates moving the data safely from moving it with fingers crossed.
Deep dive: why the parallel-run isn't optional, and why the cause is fixed
Two ideas of this step deserve development, because they're the ones haste usually skips.
The parallel-run is the difference between catching the bug before or after. The backfill's two bugs —skipping inactive, dropping a digit— have something in common that makes them dangerous: they don't throw any exception. The backfill "finished without error"; it copied rows, it didn't blow up. A team that confuses "finished without error" with "it's fine" would go straight from the backfill to the read-switch, and there the bugs become visible to the users: the webcam-hd would disappear from the catalog (a complaint), and the mech-kbd would sell for $7.99 instead of $79.99 (a loss). The parallel-run is the only thing that catches this kind of error, because it doesn't trust that the backfill "didn't blow up": it compares the result against the source, row by row.
old_store (source) new_store (buggy backfill) parallel_run
─────────────────── ────────────────────────── ─────────────
1 SSD 1TB 8999 Y --> 1 SSD 1TB 8999 True OK
2 USB-C Hub 3499 Y --> 2 USB-C Hub 3499 True OK
3 Webcam HD 5999 N --> (skipped by the bug) MISSING_IN_NEW
4 Mech Kbd 7999 Y --> 4 Mech Kbd 799 True FIELD_MISMATCH (799!=7999)
5 Mouse Pro 2499 Y --> 5 Mouse Pro 2499 True OK
the parallel_run compares and does NOT let
the read_switch through with 2 gaps
Notice that the parallel-run distinguishes two types of discrepancy, and that distinction matters: MISSING_IN_NEW (a row that didn't arrive) points to a coverage problem of the backfill (it skipped something); FIELD_MISMATCH (a row that arrived wrong) points to a translation problem (a field was copied incorrectly). Knowing the type leads you to the cause: the missing row sends you to check what the backfill filters; the field that doesn't match sends you to check that field's translation. The parallel-run doesn't just say "there's a problem"; it says what kind of problem and in which row, which is what makes the reconciliation targeted.
The cause is fixed, not the symptom. When the parallel-run reports the 2 discrepancies, the temptation is to fix them by hand: insert the missing webcam-hd, correct the mech-kbd's price. It lowers the counter immediately, yes, but it leaves the bugs alive: if the backfill is re-run, or if there are other inactive ones and other prices, they fail again. That's why this example's reconciliation fixes the cause —the filter that skips inactive ones and the translation that drops the digit— and re-runs the backfill, which now covers all the inactive ones and translates all the prices correctly in one shot. A real catalog can have hundreds of inactive products; copying the webcam-hd by hand would leave the others missing. Fixing the cause covers them all and leaves the backfill correct for the future. This is the discipline that makes the parallel-run return to zero for real, not that the counter drops while the bug is still there.
A nuance about reversibility: until the read-switch, and even for a while after, the migration is reversible. With the dual-write on, the old stays synchronized, so if something goes wrong after the switch, a change of read_source back to old returns you to the old store, which was hot the whole time. That's why the read-switch doesn't turn off the dual-write immediately: the old is the safety net of the moment of maximum risk (the first time reading from the new live). Turning off the dual-write and retiring the old comes later, when the confidence in the new is total —part of lesson 7's done criterion—.
Common mistakes
Skipping the parallel-run "because the backfill finished without error". What happens: the team, in a hurry, does the dual-write and the backfill, and since neither threw an exception, goes straight to the read-switch. Why it happens: "finished without error" gets confused with "it's fine", and the parallel-run is seen as an optional double-check step. How to spot it: in this project's cycle, this would be jumping from step 1 (backfill) to step 3 (read-switch) without step 2. How to fix it: the backfill's two bugs (skipping inactive, dropping a digit) are exactly the kind of error that doesn't throw an exception and that only the parallel-run catches. Without it, the read-switch would have moved the reads to the new with the webcam-hd absent and the mech-kbd at $7.99 —discovered through a complaint and a loss, not through a comparison—. The parallel-run isn't optional: it's the difference between catching the bug in step 1 (no harm) or in production (with affected users).
Reconciling the symptom and not the cause. What happens: the parallel-run reports the 2 discrepancies, and the team inserts the webcam-hd and corrects the mech-kbd's price by hand, instead of fixing the backfill's bugs. Why it happens: copying/correcting two rows lowers the counter immediately; investigating the bugs is slower. How to spot it: the 2 discrepancies disappear, but if there are other inactive ones or other affected prices (or if the backfill is re-run), they fail again —the bugs are still alive—. How to fix it: fix the cause (the inactive filter, the price translation) and re-run the backfill, which now covers all the inactive ones and all the prices in one shot. A real catalog with hundreds of inactive ones would leave most missing if you only copied the webcam-hd by hand. The fixed cause covers them all and leaves the backfill correct.
Switching the read to the new with open discrepancies. What happens: the team sees that "almost everything matches" (3 of 5 rows right) and does the read-switch anyway, thinking it'll fix the remaining 2 later. Why it happens: the read-switch is the visible and satisfying step; waiting for the parallel-run to come out zero feels like a delay. How to spot it: the reads already come from the new while the parallel-run still reports discrepancies; the users start seeing the bad rows. How to fix it: the hard rule is a single one —you don't switch a read to the new while the discrepancies aren't zero—. "Almost matches" isn't matches: moving the reads with 2 open gaps means serving the absent webcam-hd and the bad price to real users. The read-switch is the reward you collect after the reconciliation, not before. Only the parallel-run at zero authorizes the switch.
Exercises
Exercise 1 — The two types of discrepancy. The parallel-run reported prod 3: MISSING_IN_NEW and prod 4: FIELD_MISMATCH. (a) What backfill bug caused each one? (b) What different cause does each type point to? (c) What would a user have seen of each one if the read-switch had been done in step 1?
See solution
(a) prod 3: MISSING_IN_NEW was caused by the bug of skipping the inactive ones (backfill_skips_inactive): the webcam-hd has act: "N", so the backfill never copied it to the new. prod 4: FIELD_MISMATCH was caused by the bug of dropping a digit when translating the price (backfill_drops_digit): the mech-kbd was copied, but with price_cents = 799 instead of 7999.
(b) MISSING_IN_NEW points to a coverage problem: the backfill left out a row (a filter that excludes something it shouldn't). FIELD_MISMATCH points to a translation problem: the row arrived, but a field was copied incorrectly. Knowing the type directs the reconciliation: the missing one sends you to check what the backfill filters; the field that doesn't match sends you to check how that field is translated.
(c) With the read-switch done in step 1, a user searching for the webcam-hd (MISSING_IN_NEW) wouldn't find it —the product would have disappeared from the catalog, even though it still existed in the old—. And a user buying the mech-kbd (FIELD_MISMATCH) would see it at $7.99 instead of $79.99 —a direct loss, a product sold at a tenth of its price—. The two bugs, invisible while reading from the old, would have become real damage the instant of the premature switch.
Exercise 2 — Cause vs symptom. The team debates how to reconcile the 2 discrepancies. Ana proposes copying the webcam-hd by hand and correcting the mech-kbd's price; Beto proposes fixing the backfill's bugs and re-running it. (a) What does each approach achieve in the short term? (b) Why is Beto's the right one? (c) What would happen with Ana's if the catalog had 300 inactive products?
See solution
(a) In the short term, both lower the counter to 0: Ana inserts the missing row and corrects the bad price; Beto fixes the filters and re-runs the backfill, which does the same but through the cause. In the immediate output, both reach "0 discrepancies".
(b) Beto's is the right one because it fixes the cause: the filter that skips inactive ones and the translation that drops the digit. That means any inactive one and any price are covered, not just the two the parallel-run reported this time. Besides, it leaves the backfill correct for the future: if it's re-run (for a re-sync, for more data), it no longer reintroduces the bugs. Ana's fixes the symptom: the two rows that failed today, leaving the bugs alive to fail again.
(c) With 300 inactive products, Ana's approach would leave 299 missing: she only copied the webcam-hd by hand, but the bug (backfill_skips_inactive) skipped all 300. The parallel-run, if it were really re-run over the whole catalog, would keep reporting 299 MISSING_IN_NEW. Fixing the cause (the filter) covers all 300 in one shot. This is exactly the danger of reconciling the symptom: it works in the two-row demo and fails in the real catalog.
Exercise 3 — The hard rule and reversibility. (a) State the read-switch's hard rule and why it exists. (b) Why doesn't the read-switch turn off the dual-write immediately? (c) If after the read-switch the new started giving problems, how would you go back, and why is it possible?
See solution
(a) The hard rule: you don't switch a read to the new store while the discrepancies aren't zero. It exists because moving the reads to the new with open discrepancies means serving bad data (missing rows, mistranslated fields) to real users. The parallel-run at zero is the only authorization for the switch; "almost matches" isn't enough —each open discrepancy is a user who'll see incorrect data the instant of the switch—.
(b) Because the read-switch is the moment of maximum risk (the first time reading from the new live), and the old, kept hot by the dual-write, is the safety net of that moment. If something goes wrong after the switch, having the old synchronized lets you go back. Turning off the dual-write immediately would remove that net exactly when it's most needed. The dual-write is turned off later, when the confidence in the new is total (part of lesson 7's done).
(c) I'd go back by changing read_source back to old: the reads would return to the old store. It's possible because the dual-write stayed on after the switch, so the old stayed synchronized (receiving every write) the whole time —it's hot, with up-to-date data, ready to serve again—. Reversibility is a property of the method: until the dual-write is turned off and the old is retired, a change of read_source undoes the switch without losing anything. The risk is bounded at each step, not concentrated in a leap of no return.
Summary and next step
In this lesson you took the method's fourth step: migrating the catalog's data from shared_db to owned_db without downtime (module 6). You executed the complete cycle: the dual-write on (the front covered), the backfill filling the new —with two real bugs: skipping inactive ones and dropping a digit—, the parallel-run comparing row by row and catching the 2 discrepancies (MISSING_IN_NEW, FIELD_MISMATCH) before the switch, the reconciliation fixing the cause down to zero, and the read-switch moving the reads to the new only with the parallel-run green. You saw, with the clinic archive that moves without closing, that the audit before the switch is what makes the move safe, and you learned the hard rule —not switching a read to the new with open discrepancies—, why the parallel-run isn't optional (it catches the errors that don't throw an exception) and why the cause is fixed and not the symptom (to cover all the cases, not just the reported ones).
Before moving on you should be able to: order the phases of a data migration (dual-write → backfill → parallel-run → reconcile → read-switch); explain why the parallel-run catches bugs that the "error-free" backfill hides; distinguish MISSING_IN_NEW from FIELD_MISMATCH and what cause each one points to; and defend the read-switch's hard rule.
Lesson 6 takes the fifth step: measuring the slice's progress (module 7). With the slice characterized, behind the facade, extracted, and with its data migrated, the question is how to know if the migration is advancing and at what pace. You're going to execute the burn-down of calls to the legacy dropping to zero —with the final stubborn stretch (the volume discount you saw in lesson 3) that only closes when the modern implements it, guided by lesson 2's golden master— and the fitness function that gives FAIL if someone adds new code to the legacy you're killing. The burn-down says how much is left; the fitness keeps the legacy from growing back behind you.
Resources
- Sam Newman, Monolith to Microservices (O'Reilly, 2019), ch. 4, "Decomposing the Database" — the map of this lesson: turning on the synchronization, loading the data, verifying, cutting the reads, and retiring the old store, all with the system alive. The comprehensive reference for migrating a monolith's data. In English.
- Martin Fowler, "Patterns of Legacy Displacement" — martinfowler.com/articles/patterns-legacy-displacement. Running the old and the new over the same data and comparing before trusting; the parallel run as a named pattern is Sam Newman's (Monolith to Microservices). The exact mechanism that catches this lesson's 2 discrepancies. In English.
- Stripe Engineering, "Online migrations at scale" — stripe.com/blog/online-migrations. The production account of the four phases (dual-write, backfill, comparison, cut) over a real, in-use table: the at-scale equivalent of this project's cycle. In English.
- Pramod Sadalage and Martin Fowler, Refactoring Databases (Addison-Wesley, 2006) — the framework of evolving and migrating a database in small, reversible, and verified steps, without downtime. The go-to book for taking this step to a real schema. In English.