Module 6: Migrating Data Without Downtime
Dual-write: writing to both stores
Overview
Lesson 2 gave you the framework —expand-contract, add before removing—. This lesson puts its first piece in motion over the real data: the dual-write. The idea is simple and it's pure "expand": while the transition lasts, every write from the app goes to both stores. You write to the old (old_db) as always, and you also write to the new (new_db), translating the row with the anti-corruption layer. You don't remove the old destination; you add the new one alongside. This way, everything written from now on is reflected in both, and the new store starts staying up to date with reality without anyone noticing the change.
The dual-write solves a very concrete problem: the moving front. While you migrate, the system doesn't stop —products come in, prices change, stock is adjusted—. If you only copied the current state of the old to the new and then moved the reads, every write that occurred during the migration would be left out of the new, and the new would serve stale data from day one. The dual-write closes that leak toward the future: from the instant it's turned on, no new write escapes the new store.
But the dual-write has an exact limit, and understanding it is half of this lesson: it only covers what happens from when it's turned on. What was already in the old before —the historical accumulated over years— the dual-write doesn't even look at. Turning on the dual-write doesn't copy the past; it only guarantees the future. That historical gap is real and it's big, and closing it is the work of the next lesson (the backfill). That's why the dual-write and the backfill are a pair: one covers the front (what's happening now), the other covers the rear (what came before), and together they leave the new complete.
Connection with the module. This lesson is the "expand" of the writes: adding the second destination without removing the first (the framework of lesson 2, instantiated). Lesson 4 (backfill) fills the historical gap the dual-write leaves open —and there you'll see why the order is first dual-write, then backfill—. Lesson 5 (parallel-run) will check that what the dual-write and the backfill wrote to the new really matches the old. Notice the boundary: here the dual-write is an in-memory function that writes to two dictionaries; in production, writing to two stores consistently (what happens if the second write fails?) is a deep problem that the CDC tools of the Data Engineering ecosystem solve in another way —reading the transaction log of the primary instead of writing twice from the app—. Here you learn the pattern; the deep dive explains that alternative.
An analogy: mail forwarding when you move
You move house. You keep receiving mail —letters from the bank, bills, magazines— and you can't afford to lose any while you make the change. The post office offers you a service: forwarding. From the day you activate it, every new letter that arrives at your old address is copied and also sent to the new. From that instant, no matter where they send it, the letter reaches you at the new house.
Notice the two things forwarding does and doesn't do, because they're exactly the ones of the dual-write:
What it does do: it covers all the new mail, from when you activate it. Once the forwarding is on, any letter someone sends you —even if addressed to the old house— also appears at the new. The "front" of your correspondence is covered: nothing new is lost. And you didn't stop receiving at the old: you still have both addresses active, just in case. It's additive, not destructive.
What it doesn't do: it doesn't move the letters that were already in your drawers. The forwarding starts operating the day you activate it and only over the mail that arrives after. All the letters you already received and stored at the old house —your years of archived correspondence— stay at the old house. The forwarding doesn't go into your drawers to copy them; it only looks forward. If you want that history at the new house, you have to carry it yourself, in boxes —and that's the backfill of lesson 4—.
And there's a detail that corresponds to the anti-corruption layer: if the new house uses a different address format (another postal code, another numbering system), the post office translates the address when forwarding. The letter leaves with the old format and arrives with the new. In the dual-write, that translation is the ACL: the write arrives in the old model (sku, price_cents) and is stored in the new (id, price_usd).
So the dual-write is mail forwarding: turn it on and, from that instant, every new write lands in both stores, translated to the format of each. But what was already archived before stays only in the old until you carry it in boxes. Let's see exactly that, executed.
Worked example: the dual-write turning on midway through the flow
We're going to run a flow of writes where the dual_write turns on midway, to see clearly what lands where. The first three writes happen with the dual-write off (they land only in the old); from the fourth on, the dual-write is on (they land in both). We include on purpose an update of a product that already existed (ssd-1tb drops in price) to see that the dual-write not only synchronizes inserts, but also changes.
old_db = {}
new_db = {}
dual_write_on = False
def to_new_model(old_row):
return {
"id": old_row["sku"],
"title": old_row["name"],
"price_usd": old_row["price_cents"] / 100,
"in_stock": old_row["stock"] > 0,
}
# --- The app ALWAYS writes to the old; and to the new ONLY if dual_write is ON. ---
def write_product(row):
old_db[row["sku"]] = row
landed = "old"
if dual_write_on:
new_db[row["sku"]] = to_new_model(row)
landed = "old + new"
return landed
# --- A write flow. The first 3 happen BEFORE dual_write. ---
events = [
("ssd-1tb", "SSD 1TB", 8999, 12),
("usb-c-hub", "USB-C Hub", 3499, 40),
("webcam", "Webcam HD", 5999, 0),
# --- dual_write turns on here ---
("hdmi-cable", "HDMI 2m", 1299, 30),
("ssd-1tb", "SSD 1TB", 8499, 12), # price update: 89.99 -> 84.99
("kbd-mech", "Mech Kbd", 7999, 5),
]
print("Write flow: dual_write turns on before write #4\n")
print(f"{'#':>2} {'dual_write':<11}{'sku':<12}{'landed in':<12}")
print("-" * 40)
for i, (sku, name, cents, stock) in enumerate(events, start=1):
if i == 4:
dual_write_on = True # <- dual_write turns on
landed = write_product({"sku": sku, "name": name, "price_cents": cents, "stock": stock})
flag = "ON" if dual_write_on else "OFF"
print(f"{i:>2} {flag:<11}{sku:<12}{landed:<12}")
print("-" * 40)
only_in_old = [sku for sku in old_db if sku not in new_db]
in_both = [sku for sku in old_db if sku in new_db]
print(f"\n Rows only in old (historical, pre-dual-write): {len(only_in_old)} -> {only_in_old}")
print(f" Rows in both (written with dual_write ON): {len(in_both)} -> {in_both}")
print("\n Notice 'ssd-1tb': its update (write #5, with dual_write ON) DID reach")
print(" the new -> new['ssd-1tb'] = 84.99. dual_write syncs what's happening NOW.")
print(f" new['ssd-1tb'].price_usd = {new_db['ssd-1tb']['price_usd']}")
print("\n But usb-c-hub and webcam (written BEFORE) are still ONLY in the old.")
print(" dual_write doesn't look back: closing that historical gap is the backfill (L4).")What to expect. When you run the file, the output is exactly this:
Write flow: dual_write turns on before write #4
# dual_write sku landed in
----------------------------------------
1 OFF ssd-1tb old
2 OFF usb-c-hub old
3 OFF webcam old
4 ON hdmi-cable old + new
5 ON ssd-1tb old + new
6 ON kbd-mech old + new
----------------------------------------
Rows only in old (historical, pre-dual-write): 2 -> ['usb-c-hub', 'webcam']
Rows in both (written with dual_write ON): 3 -> ['ssd-1tb', 'hdmi-cable', 'kbd-mech']
Notice 'ssd-1tb': its update (write #5, with dual_write ON) DID reach
the new -> new['ssd-1tb'] = 84.99. dual_write syncs what's happening NOW.
new['ssd-1tb'].price_usd = 84.99
But usb-c-hub and webcam (written BEFORE) are still ONLY in the old.
dual_write doesn't look back: closing that historical gap is the backfill (L4).
Read the table and the two lists at the end, because there is the whole lesson.
The landed in column tells the story. Writes 1, 2, and 3 happened with the dual_write OFF: they landed only in the old (old). Write 4 onward happened with the dual_write ON: each one landed in both (old + new). The turn-on point divides the flow in two: before, only the old; after, both. That's the forwarding activating: everything that arrives after is copied to the new house.
Notice write 5, the update of ssd-1tb. ssd-1tb had already been written in write 1 (with the dual-write off, at 89.99), but its price update to 84.99 happened with the dual-write already on. Result: that change did reach the new —new['ssd-1tb'].price_usd = 84.99—. The dual-write doesn't distinguish between inserts and changes: any write, while it's on, lands in both. It synchronizes the present state.
Now read the two lists at the end, which are the heart of the lesson:
- Rows only in the old:
['usb-c-hub', 'webcam']. They're the ones written in writes 2 and 3, before turning on the dual-write. They never reached the new, and the dual-write —which only looks forward— won't carry them. They're the history left behind: the mail in your drawers that the forwarding doesn't touch. - Rows in both:
['ssd-1tb', 'hdmi-cable', 'kbd-mech'].ssd-1tbreached the new through its update (write 5);hdmi-cableandkbd-mechby being new inserts (writes 4 and 6). All written with the dual-write on.
The conclusion, in one sentence: the dual-write closes the leak toward the future, but leaves the gap of the past open. usb-c-hub and webcam prove that turning on the dual-write isn't enough —the new still doesn't have everything—. Filling that gap is the backfill, lesson 4. And there's a reason the dual-write goes first and the backfill after: you'll see in lesson 4 that this order is what guarantees no write is lost in the transition.
Deep dive: what the dual-write guarantees and what it doesn't
The dual-write has a precise guarantee and a couple of subtleties worth facing head-on, because in production they're what decides whether it works.
The guarantee: zero leaks forward. From the instant the dual-write is on, there's no write that lands in the old and not in the new. The "moving front" of the data is covered. This is what makes it possible, later, for the new to take over: it's not falling behind while you migrate.
The non-guarantee: the history. You already saw it: the dual-write doesn't copy the earlier. It's important not to confuse "the dual-write is on" with "the new is complete". The first is true from when you turn it on; the second only after the backfill. Confusing them leads to doing the read-switch too soon, with the new full of historical gaps.
The order of the two writes. In the example, write_product writes first to the old and then to the new. That order matters: the old is the source of truth during the whole transition (it's where it's still read from), so its write is the one that can't fail. The write to the new is "extra" for now —the new doesn't yet serve reads—, so if of the two one is going to go first, the old's goes. You write to the truth first, to the copy after.
The hard problem: what happens if the second write fails? Here's the uncomfortable part of the dual-write, the one the in-memory example hides. In production, writing to two stores isn't atomic: the write to the old may succeed and the write to the new may fail (a network drop, a timeout). Now the old has the data and the new doesn't —an inconsistency—. What do you do? You have options, all with costs: retry the write to the new, queue it for later, or record the failure and let the parallel-run detect it later. None is free. This is precisely the reason why, at scale, many teams don't do dual-write from the app, but use change data capture (CDC): instead of the app writing twice, a tool reads the transaction log of the old store (where all the writes that did succeed are already recorded) and replays it in the new. This way there's only one write from the app (to the old), and the copy to the new is derived from a reliable record. The dual-write from the app is simpler to understand and to simulate —that's why we use it here—, but its fragility against partial failures is real, and CDC is the production answer. That tool and its operation belong to the Data Engineering ecosystem; the pattern they implement is the one you execute in this lesson.
The parallel-run as a net under the dual-write. Precisely because the dual-write can fail partially (or the ACL can translate wrong), you don't trust it blindly. The parallel-run of lesson 5 reads again from both and compares: if a write to the new was lost or stored wrong, the comparison exposes it. The dual-write does the work; the parallel-run verifies it. Never one without the other.
Common mistakes
Believing that turning on the dual-write completes the new store. What happens: the team turns on the dual-write, sees the new writes landing in both, and assumes the new "already has the data", ready to read from it. Why it happens: it's easy to see the flow of new writes filling the new and forget the huge history that never went through there. How to spot it: the new is missing all the records that haven't been written since the dual-write was turned on —for a catalog, the products nobody edited recently; for orders, everything before the turn-on date—. How to fix it: remember the pair. The dual-write covers the front; the backfill covers the history. The new is complete only when both ran, and the proof that it's complete is the green parallel-run (lesson 5), not "the dual-write has been on for a while". Turning on the dual-write is the beginning of the expand, not its end.
Ignoring the partial failures of the second write. What happens: the dual-write is implemented as "write to the old, write to the new", without considering what happens if the second fails; when it fails in production (network, timeout), the new is left with one record less than the old and nobody notices. Why it happens: in an in-memory example the two writes never fail, so the problem is invisible until production. How to spot it: the parallel-run starts reporting discrepancies that "appear on their own" —rows where the old has the new value and the new has the old, because its update was lost—. How to fix it: explicitly decide the policy on a failure of the second write (retry, queue, or record to reconcile), and —above all— don't trust the dual-write as the only source of synchronization: the parallel-run is there to catch exactly these leaks. At scale, consider CDC instead of dual-write from the app, precisely to eliminate the partial failure (a single write, the copy derived from the log).
Writing to the new first and the old after. What happens: through carelessness, the dual-write writes to the new store before the old; if the write to the old fails, the new has a record the old —the source of truth, where it's still read from— doesn't have. Why it happens: the order of two lines seems irrelevant. How to spot it: a user writes something, the write to the old fails silently, the read (which still comes from the old) doesn't show their change, but the new does have it —a confusing incoherence—. How to fix it: during the transition, the old is the source of truth and its write is the priority: it goes first, and it's the one that can't be lost. The new's goes after, as the copy. When in lesson 7 the read-switch is done and the new becomes the source of truth, that order will invert —but that's at the end, not now—.
Exercises
Exercise 1 — Read where each write landed. In the output, usb-c-hub ended up "only in old" but kbd-mech ended up "in both", even though both are catalog products. (a) What differentiates them? (b) Why did the update of ssd-1tb (write 5) reach the new, if ssd-1tb had been created before the dual-write? (c) If after turning on the dual-write someone wrote usb-c-hub again (any update), would it appear in the new?
See solution
(a) What differentiates them is the moment they were written relative to the dual-write's turn-on. usb-c-hub was written in write 2, with the dual-write off: it landed only in the old. kbd-mech was written in write 6, with the dual-write on: it landed in both. It's not a property of the product, but of the instant of its write.
(b) Because the dual-write acts on each write while it's on, regardless of whether the record already existed. The update of ssd-1tb (dropping the price to 84.99) is a write that happened at step 5, with the dual-write already ON, so it was applied to both stores. What matters isn't when the record was created, but when this write happened.
(c) Yes. If usb-c-hub receives any new write with the dual-write on, that write would land in both stores and usb-c-hub would appear in the new. In fact, if all the historical records received a write after the turn-on, the dual-write would end up filling the new completely... but you can't count on that: there are records nobody edits for years (a discontinued product, an old order). That's why the backfill is necessary: it guarantees the history reaches the new without depending on someone touching it again.
Exercise 2 — Mail forwarding. With the analogy of forwarding when you move: (a) what part of the mail does the forwarding cover and what part not? (b) translate that to "what the dual-write covers and what not"; (c) what action of the move corresponds to the backfill, and why is it necessary in addition to the forwarding?
See solution
(a) The forwarding covers all the mail that arrives after activating it: every new letter is copied to the new house. It does not cover the letters that were already in your drawers at the old house —those stay where they are—. The forwarding looks only forward, from the day of activation.
(b) Same as the dual-write: it covers all the writes that happen from when it's turned on (they land in both stores), but it does not cover what was written before turning it on (it stays only in the old). The dual-write synchronizes the front, not the history.
(c) The backfill corresponds to carrying, yourself, in boxes, the archived letters from the old house to the new. It's necessary in addition to the forwarding because the forwarding will never move those old letters —it only looks forward—; if you want the complete history at the new house, someone has to go to the drawers, take out the archived, and carry it. In data: the backfill copies to the new the historical records that the dual-write, by definition, doesn't touch. Without the backfill, the new would have only the recent mail and would be missing the whole archive.
Exercise 3 — The partial failure. In production, the dual-write writes to the old and then to the new, but the write to the new fails from a network timeout in one of every thousand writes. (a) In what state is that row left (old vs new)? (b) Why does the in-memory example of this lesson never show this problem? (c) What mechanism of the module will end up detecting that out-of-sync row, and why does CDC avoid it at the root?
See solution
(a) It's left out of sync: the old has the new value (its write succeeded) and the new has the previous value or doesn't have the row (its write failed). Since during the transition it's read from the old, the user sees the correct data, but the new fell behind on that row —a silent inconsistency that throws no visible error—.
(b) Because in memory the two writes to Python dictionaries never fail: there's no network, no timeouts, no unavailability. The example simulates the pattern (write to both), but not the infrastructure (two real stores that can fail independently). The partial failure only appears when the stores are separate systems that can be up or down on their own.
(c) The parallel-run (lesson 5) will end up detecting it: on reading from both and comparing, that row will come out as VALUE_MISMATCH (or MISSING_IN_NEW if the write failed completely), and the reconciliation (lesson 6) will fix it by re-migrating it. CDC avoids it at the root because it eliminates the second write from the app: instead of the app writing twice (and one being able to fail), the app writes only once to the old, and a tool reads the transaction log of the old —where only the writes that did succeed are recorded— and replays them in the new. There's no "second write" that can fail independently: the copy is derived from a reliable record of what was really written.
Summary and next step
In this lesson you put in motion the first piece of the migration over the data: the dual-write. You saw, with mail forwarding when you move, that turning on the dual-write makes each new write land in both stores —translated by the ACL to the format of each—, closing the leak toward the future. And you saw its exact limit, executed: the dual-write doesn't look back. usb-c-hub and webcam, written before the turn-on, stayed only in the old; only what was written with the dual-write on (ssd-1tb, hdmi-cable, kbd-mech) reached the new. The dual-write synchronizes the front, not the history.
Before moving on you should be able to: explain what the dual-write guarantees (zero leaks forward) and what it doesn't (the history); read a table of writes and say which landed in both and why; argue why the write to the old goes first (it's the source of truth); and recognize the problem of the partial failure of the second write and why CDC solves it at scale.
Lesson 4 closes the gap the dual-write leaves open: the backfill. You're going to fill the new store with the historical records —usb-c-hub, webcam, and all the others left behind—, and you're going to discover its two hard rules, executed: don't overwrite the fresh writes the dual-write already put (a careless backfill overwrites a just-updated price with its old value) and the idempotence (running it twice doesn't corrupt or duplicate). And you'll see, finally with the code in hand, why the backfill goes after turning on the dual-write and not before: that order is what guarantees no write is lost in the transition.
Resources
- Sam Newman, Monolith to Microservices (O'Reilly, 2019), ch. 4, "Decomposing the Database" — the section on synchronizing data between the monolith and the new service during the migration, including the pattern of writing to both and its consistency risks. The direct reference of this lesson. In English.
- Martin Fowler, "ParallelChange" — martinfowler.com/bliki/ParallelChange.html. The dual-write is the "expand" of the writes: adding the second destination without removing the first. The pattern entry that frames it. In English.
- Debezium, "Documentation: Change Data Capture" — debezium.io/documentation. The reference CDC tool: how to read a database's transaction log and publish its changes, the production alternative to dual-write from the app. To understand what's underneath when this is done at scale (Data Engineering ecosystem). In English.
- Chris Richardson, "Pattern: Database per service" — microservices.io/patterns/data/database-per-service.html. The context of why an extracted service needs its own data, and the patterns to keep it synchronized during the transition from a shared database. In English.