Module 6: Migrating Data Without Downtime

The read-switch and turning off the dual-write

Next →

Overview

You reached the final cut. The dual-write covered the front, the backfill covered the history, the parallel-run compared, and the reconciliation brought the discrepancies to zero —sustained—. The new store, finally, tells the truth in all the rows and stays matched while the system operates. Only now, with that evidence in hand, is the read-switch executed: moving the reads from the old to the new. And then, in an order that matters, the dual-write is turned off and the old is retired.

The final cut isn't a single step, it's three, and the order between them is what makes the lesson:

  1. read-switch: the reads now come from the new. The old stays warm —the dual-write is still on, writing to it—, because it's your backup net: if something goes wrong with the newly-promoted source of truth, you can return the reads to the old in an instant.
  2. turn off the dual-write: when you've been reading from the new long enough without problems, you stop writing to the old. The old is frozen.
  3. retire the old: when nobody reads or writes it, you turn it off and delete it.

The hard rule of this lesson is the order: read-switch first (with the old still alive as a net), turn off the dual-write after, retire the old at the end. Inverting the first two steps is a concrete and dangerous mistake —you're going to see it executed—: if you turned off the dual-write before the read-switch, the old would stop receiving writes while the reads still come out of it, and you'd be serving stale data to users. The order isn't an aesthetic preference; it's what avoids that gap.

And there's a closing almost nobody remembers, and that this module insists on not forgetting: turning off the dual-write for real. It's easy to do the read-switch, see the new serve the reads well, and leave the dual-write on "just in case" —writing forever to an old store nobody reads anymore—. That's the migration that never ends: two systems maintained for eternity, double the cost and complexity, without anyone reaping the prize of having migrated. The final cut ends when the old is retired, not when the reads moved.

Connection with the module. This lesson consumes the sustained zero of lesson 6 and executes the closing. It's the "migrate + contract" of the reads that lesson 2 (expand-contract) anticipated: moving the readers to the new (migrate) and retiring the old (contract), in that order. Notice the boundary: the read-switch of this module moves the reads of data; in module 3, the strangler moved the reads and writes of traffic with a facade —the symmetry is total, and the "retiring the legacy" of M3 is the sibling of the "retiring the old" here—. Measuring the progress of all this (the burn-down of calls to the old until zero, avoiding the eternal migration) is module 7, which takes this cut and turns it into a metric.

An analogy: switching from the old radar to the new in the control tower

In an air control tower, the controllers watch the planes on a radar screen. The tower is going to replace its old radar with a new one. The two radars track the same planes —they're installed in parallel, both on and updating in real time—; the only question is which screen the controllers look at to make decisions. Today they look at the old radar's.

The change is made with extreme care, because there are lives at stake, and the order is sacred:

  1. The controllers switch to looking at the new screen (the read-switch). From this moment, their decisions are based on the new radar. But —and this is the important thing— the old radar stays on and updating. Nobody turns it off. It's the backup net: if the new screen flickers, freezes, or shows something odd, the controllers go back to looking at the old in a second, without having lost track of any plane.

  2. They keep both radars alive for a good while (the dual-write stays on). For days, the new screen is the one watched, but the old is kept fed, ready to take over. They observe: does the new track everything well? does it match what the old showed? does any plane disappear from one and not the other? Only when there's full confidence in the new do they move to the next step.

  3. They stop maintaining the old radar (turn off the dual-write). Nobody looks at it anymore and the new proved reliable; they stop feeding it. The old radar is frozen, with the last image it had.

  4. They dismantle the old radar (retire). It's turned off completely and removed from the tower. It no longer takes up space, consumes energy, or confuses anyone.

Now, the mistake the order avoids, and that in a control tower would be catastrophic: what would happen if they stopped feeding the old radar while the controllers are still looking at it? The old screen would freeze —it would keep showing the last position of each plane, but no longer update—. The controllers would be making decisions on a stale image: they'd believe a plane is still where it was five minutes ago, when in reality it already moved. It's exactly the danger of the inverted order in data: turning off the dual-write (stopping feeding the old) before the read-switch (before the controllers look at the new) makes the reads come out of a store that no longer updates —stale data served as if it were fresh—.

That's why the order is: first you look at the new screen (read-switch), with the old still alive as backup; only after do you stop feeding the old (turn off dual-write); and at the end you dismantle it (retire). You never stop feeding the one you're still looking at. Let's execute exactly this sequence.

Worked example: the final cut, step by step

We start with old_db and new_db already synchronized and the parallel-run green. There are two write knobs —write_to_old and write_to_new— and one read knob —read_source—. The dual-write on means the two write knobs are True. We're going to execute the six steps of the cut and observe the read ssd column (where the price a user reads comes from) and the dual_write column at each step.

# --- Two write knobs + where we read from. dual_write = write to both.
write_to_old = True
write_to_new = True         # dual_write ON: writes to both
read_source  = "old"        # reads still come from the old

def to_new_model(old_row):
    return {"id": old_row["sku"], "title": old_row["name"],
            "price_usd": old_row["price_cents"] / 100, "in_stock": old_row["stock"] > 0}

old_db = {}
new_db = {}

def write_product(row):
    landed = []
    if write_to_old:
        old_db[row["sku"]] = row;                 landed.append("old")
    if write_to_new:
        new_db[row["sku"]] = to_new_model(row);   landed.append("new")
    return "+".join(landed) if landed else "(nothing)"

def read_price(sku):                              # reads from the ACTIVE store
    if read_source == "new":
        return round(new_db[sku]["price_usd"] * 100)
    return old_db[sku]["price_cents"]

def dual_state():
    return "ON" if (write_to_old and write_to_new) else "OFF"

# --- Starting state: old and new already synchronized, parallel_run green. ---
for sku, name, cents, stock in [("ssd-1tb", "SSD 1TB", 8499, 12),
                                 ("kbd-mech", "Mech Kbd", 7999, 5)]:
    old_db[sku] = {"sku": sku, "name": name, "price_cents": cents, "stock": stock}
    new_db[sku] = to_new_model(old_db[sku])

print("The final cut, step by step (old and new already match, parallel_run green)\n")
print(f"{'step':<24}{'dual_write':<11}{'reads':<7}{'read ssd':>9}   observation")
print("-" * 82)

# STEP 1: parallel_run green -> the switch is authorized (reads still from old).
print(f"{'1 parallel_run green':<24}{dual_state():<11}{read_source:<7}{read_price('ssd-1tb'):>9}   discrepancies=0 -> authorizes")

# STEP 2: read_switch. The reads now come from the new. dual_write STAYS ON.
read_source = "new"
print(f"{'2 read_switch':<24}{dual_state():<11}{read_source:<7}{read_price('ssd-1tb'):>9}   reads moved to new")

# STEP 3: a write arrives. With dual_write ON it lands in BOTH (old stays warm).
landed = write_product({"sku": "ssd-1tb", "name": "SSD 1TB", "price_cents": 8299, "stock": 12})
print(f"{'3 write ssd 82.99':<24}{dual_state():<11}{read_source:<7}{read_price('ssd-1tb'):>9}   landed in {landed} (old as backup)")

# STEP 4: turn off dual_write. The writes stop going to the old.
write_to_old = False
print(f"{'4 dual_write OFF':<24}{dual_state():<11}{read_source:<7}{read_price('ssd-1tb'):>9}   no longer writing to old")

# STEP 5: another write arrives. With dual_write OFF it lands ONLY in the new; old frozen.
landed = write_product({"sku": "kbd-mech", "name": "Mech Kbd", "price_cents": 7499, "stock": 5})
print(f"{'5 write kbd 74.99':<24}{dual_state():<11}{read_source:<7}{read_price('ssd-1tb'):>9}   landed in {landed} (old NO)")

# STEP 6: retire the old. Nobody reads or writes it anymore.
old_kbd_stale = old_db["kbd-mech"]["price_cents"]     # stayed frozen at 79.99
new_kbd_fresh = round(new_db["kbd-mech"]["price_usd"] * 100)
old_db = None
print(f"{'6 retire old':<24}{dual_state():<11}{read_source:<7}{read_price('ssd-1tb'):>9}   old_db deleted")
print("-" * 82)

print("\n  Why the ORDER matters. In step 5 the old did NOT receive the kbd update:")
print(f"    old['kbd-mech'] stayed at {old_kbd_stale} (79.99, STALE); new has {new_kbd_fresh} (74.99).")
print("  Since we already read from new (step 2), that stale old is harmless: nobody reads it.")
print("  If you had turned off dual_write BEFORE the read_switch, that read of the old")
print("  would have returned a stale 79.99 to a user. That's why: read_switch FIRST,")
print("  turn off dual_write AFTER, retire the old AT THE END.")

What to expect. When you run the file, the output is exactly this:

The final cut, step by step (old and new already match, parallel_run green)

step                    dual_write reads   read ssd   observation
----------------------------------------------------------------------------------
1 parallel_run green    ON         old         8499   discrepancies=0 -> authorizes
2 read_switch           ON         new         8499   reads moved to new
3 write ssd 82.99       ON         new         8299   landed in old+new (old as backup)
4 dual_write OFF        OFF        new         8299   no longer writing to old
5 write kbd 74.99       OFF        new         8299   landed in new (old NO)
6 retire old            OFF        new         8299   old_db deleted
----------------------------------------------------------------------------------

  Why the ORDER matters. In step 5 the old did NOT receive the kbd update:
    old['kbd-mech'] stayed at 7999 (79.99, STALE); new has 7499 (74.99).
  Since we already read from new (step 2), that stale old is harmless: nobody reads it.
  If you had turned off dual_write BEFORE the read_switch, that read of the old
  would have returned a stale 79.99 to a user. That's why: read_switch FIRST,
  turn off dual_write AFTER, retire the old AT THE END.

Read the table step by step, following the columns reads (where it's read from) and dual_write (whether it's still written to the old).

Step 1 — The parallel-run is green (discrepancies = 0). The reads still come from the old (reads = old), the dual-write is ON. The price read of ssd-1tb is 8499. This is the state lesson 6 handed us: new validated, ready, but not yet in use for reads. The green zero authorizes the switch.

Step 2 — The read_switch. reads goes to new: the reads now come from the new store. The price read is still 8499 —identical to the old's, because they're synchronized; that's why the switch is transparent for the user—. Notice: the dual_write is still ON. We turn off nothing of the old; we only change where we read from. The old stays warm, like the old radar that stays on: it's the backup net. If something went wrong with the new now, a single change (read_source = "old") returns the reads to the old, which is up to date.

Step 3 — A write arrives while the read-switch already happened but the dual-write is still ON: ssd-1tb drops to 82.99. It landed in old+new (both): the user reads it from the new (82.99, updated), and the old also received it, staying warm and up to date as backup. This is the post-switch coexistence phase: you read from the new, but the old keeps mirroring just in case.

Step 4 — Turn off the dual_write. write_to_old goes to False; now dual_write shows OFF. From here, the writes stop going to the old. We do it now, not before, because we've been reading from the new without problems for a while and we trust it. The old is going to start freezing.

Step 5 — Another write arrives with the dual-write already OFF: kbd-mech drops to 74.99. It landed only in new —"(old NO)"—. The old didn't receive it: old['kbd-mech'] stayed at its previous value (7999). The old is now stale for kbd-mech. But notice: that harms nobody, because the reads come from the new (step 2), which does have the 74.99. The stale old is harmless because nobody reads it anymore.

Step 6 — Retire the old. old_db = None: we delete it. Nobody reads it (reads = new since step 2) or writes it (dual_write = OFF since step 4), so deleting it affects nothing. The price of ssd-1tb is still read fine (8299) from the new. The migration ended: the old doesn't exist.

And the final block executes the lesson of the order with numbers. In step 5, the old went stale for kbd-mech (7999 instead of 7499). Since we read from the new, it doesn't matter —nobody sees that 7999—. But imagine the inverted order: if you had turned off the dual-write (step 4) before the read-switch (step 2), the reads would still come out of the old, and that read of kbd-mech would return stale 7999 to a real user, when the true price is 7499. The old stopped updating while it was still the source of the reads: the frozen radar screen. That's the gap the correct order —read-switch first, turn off dual-write after— eliminates.

Deep dive: the order, the backup net, and the closing you don't forget

Why the old stays warm after the read-switch. It might seem that, once the reads come from the new, the old is useless and can be turned off immediately. But the read-switch is the moment of maximum risk of the whole migration: it's the first time the new is the source of truth for real reads, in production, with real traffic. If there's a problem the parallel-run didn't catch —a load the new can't handle, an edge case—, you want to be able to go back in seconds. Keeping the dual-write on after the read-switch keeps the old up to date, so the rollback is trivial: you change read_source back to old and the old, which never stopped updating, takes over without having lost anything. Turning off the dual-write too soon burns that net: as soon as the old freezes, it's no longer a valid rollback (it would be stale). That's why the sequence leaves a margin —you read from the new with the old still alive— before cutting the dual-write.

The complete order, and the symmetry of the writes. Notice a subtle detail of the example: during the transition, the write to the old went first (it was the source of truth, lesson 3). After the read-switch, the new is the source of truth, and the one that can't fail is it. "Turning off the dual-write" formalizes that transition: you stop writing to the old because it's no longer anyone's truth. The cut of reads and the cut of writes are staggered on purpose:

                 reads          writes
                 ─────          ──────────────
before the cut   old            old (truth) + new (copy)     <- dual_write ON
read_switch      new            old + new                    <- dual_write ON, old warm
dual_write OFF   new            new                          <- old frozen, net removed
retire           new            new                          <- old deleted

The reads change first (read_switch), the writes to the old are cut after (dual_write OFF), and the old is deleted at the end. Each row of that table is reversible until the second-to-last: while the old is warm (dual_write ON), you can return the reads to the old. As soon as you turn off the dual-write, the old starts aging and the rollback stops being free —that's why that step is only taken with earned confidence—.

Retiring means deleting, not "leaving it just in case". Step 6 does old_db = None: it deletes the old. In a real system, retiring means dropping the table, turning off the database, removing the code that touched it. It's not "leaving it there frozen indefinitely". An old store left "just in case" has real costs: it takes up space, someone has to back it up and maintain it, it confuses whoever reads the code ("is this table used or not?"), and it's a permanent temptation for some component to read it again by mistake. The prize of the migration —a simpler system, with one store instead of two— is only reaped when the old truly disappears. Just as the strangler of module 3 doesn't end until retiring the legacy, the data migration doesn't end until deleting the old.

The mistake of forgetting to turn off the dual-write. It's the most common silent failure of the cut. The team does the read-switch, sees the new serve well, and... leaves the dual-write on forever. The system ends up writing every datum to two stores indefinitely, when it only reads from one. The costs pile up: every write pays double (two stores), the old keeps growing without anyone reading it, and the complexity of "keeping both synchronized" becomes permanent instead of temporary. Worse: nobody decides to leave it that way; the cut simply "was considered finished" at the read-switch, and steps 4 to 6 were never executed. The rule is clear: the final cut has four steps (read-switch, turn off dual-write, wait, retire), and ends in the retirement. A cut that stops at the read-switch is a half-done migration disguised as complete. Module 7 measures exactly this —that the old actually gets turned off— so it doesn't stay on by inertia.

Common mistakes

Turning off the dual-write before the read-switch (the inverted order). What happens: the team, thinking "we already validated the new, let's stop writing to the old", turns off the dual-write while the reads still come from the old —which starts freezing and serving stale data—. Why it happens: the two steps (turn off dual-write, move reads) look like "finishing using the old", and the order between them seems interchangeable. How to spot it: stale reads appear —users who see an old price or stock— right after turning off the dual-write, and the old (still the source of the reads) doesn't reflect the recent writes. How to fix it: read-switch first, turn off dual-write after, without exception. While the reads come from the old, the old has to keep receiving writes (dual-write ON), or it'll serve frozen data. Only when the reads already come from the new is it safe to stop feeding the old. Never stop feeding the store you still read —the radar you still watch—.

Turning off the dual-write too soon after the read-switch (burning the backup net). What happens: the team does the read-switch and, minutes later, turns off the dual-write —so that when a problem with the new appears, it can no longer go back to the old, which went stale—. Why it happens: reaching the end feels close, and keeping the dual-write "a little longer" seems wasteful. How to spot it: very little time passed between the read-switch and turning off the dual-write (minutes, not days); when an incident arises in the new, the rollback to the old is no longer viable because the old froze. How to fix it: keep the dual-write on for a prudent period after the read-switch —enough to trust that the new holds up to the real traffic—. During that period the old stays up to date and the rollback is free: a read_source change and you're back. Turning off the dual-write is irreversible in practice (the old ages immediately), so it's done only with earned confidence, not out of haste.

Ending at the read-switch and not retiring the old. What happens: the reads moved to the new, everything works, and the project is declared finished —the dual-write stays on, the old stays there, forever—. Why it happens: the read-switch is the visible and satisfying moment (the new already serves!); turning off the dual-write and deleting the old are work without visible reward and with a bit of fear. How to spot it: months after the read-switch, the system still writes to both stores and the old is still deployed, even though nobody reads it. How to fix it: the cut ends in the retirement, not at the read-switch. Define steps 4-6 (turn off dual-write, wait, delete the old) as part of the project from the start, with a date. The prize of the migration —a single store, half the complexity, the end of the double write cost— is only reaped when the old disappears. Leaving it "just in case" is paying the cost of the migration without reaping its benefit: the worst of both worlds. Module 7 measures this closing so it doesn't stay half-done.

Exercises

Exercise 1 — Reconstruct the danger of the inverted order. In the example, step 5 wrote kbd-mech at 74.99 only in the new, leaving the old at 79.99. (a) Why was that stale old harmless in the correct sequence? (b) If the dual-write had been turned off before the read-switch, what would a user querying kbd-mech have read? (c) What general principle does this sum up?

See solution

(a) It was harmless because, by step 5, the reads already came from the new (the read-switch happened in step 2). The old had kbd-mech stale at 79.99, but nobody reads from the old, so that frozen value reaches no user. A stale store only does harm if someone reads it; once the reads are moved to the new, the old can age peacefully.

(b) It would have read 79.99, the stale value. With the inverted order, the reads would still come from the old, but the dual-write would already be off, so the write of kbd-mech at 74.99 would have landed only in the new —which nobody reads— and the old would have stayed at 79.99. The user would see an outdated price as if it were the current one: a stale datum served as fresh.

(c) The principle: never stop feeding the store you still read from. While the reads come from the old, the old has to keep receiving writes (dual-write ON). Only when the reads already come from the new is it safe to stop feeding the old. That's why the order is read-switch first (move the reads), turn off dual-write after (stop feeding the one nobody reads anymore).

Exercise 2 — The rollback window. After the read-switch, the team keeps the dual-write on for three days before turning it off. (a) What's that three-day window for? (b) If on day 2 the new starts having performance problems, how do you go back to the old, and why is it free? (c) Why does turning off the dual-write "make the cut irreversible", in practice?

See solution

(a) It serves as a rollback window: it's a period in which the reads already come from the new (to test it with real traffic) but the old stays up to date (dual-write ON), ready to take over if the new fails. It's the phase of maximum caution: you trust the new enough to read from it, but not enough to burn the backup net. The three days give time for problems the parallel-run didn't catch to appear (load, edge cases, traffic spikes).

(b) You go back to the old with a single change: read_source = "old". Since the dual-write was on the whole time, the old received all the writes of those two days and is completely up to date —it lost nothing—. That's why the rollback is free: there's no data to recover or resynchronize; the old already has everything, you just have to point the reads back at it. It's instant and lossless.

(c) Because in the instant you turn off the dual-write, the old stops receiving writes and starts aging. Every new write lands only in the new; the old falls further and further behind. Within a few minutes, returning the reads to the old would already mean serving stale data (it would be missing all the writes after the turn-off). In theory you could re-synchronize the old, but that's as expensive as a new migration. In practice, turning off the dual-write burns the backup net: that's why it's done only with earned confidence, at the end of the rollback window, not at the start.

Exercise 3 — The migration that doesn't end. A team did the read-switch six months ago. The new serves all the reads without problems. But the dual-write is still on and the old is still deployed. (a) What costs is the team paying for not having finished the cut? (b) What steps are they missing? (c) Why is it "the worst of both worlds"?

See solution

(a) It's paying: the double write cost (every write goes to two stores, even though it only reads from one); the maintenance of the old (backups, monitoring, space, patches for a store nobody reads); complexity and confusion (the code keeps both synchronized for no reason, and whoever reads it doesn't know if the old is used); and the risk that some component reads the old again by mistake, or that the divergence between the two causes bugs. All that to sustain a store that no longer contributes anything.

(b) They're missing steps 4-6 of the cut: turn off the dual-write (stop writing to the old, once it's confirmed the new is reliable), wait the prudent period, and retire the old (drop the table, turn off the database, remove the code that touched it). The cut stopped at the read-switch (step 2) and never reached the retirement (step 6).

(c) Because the team paid the full cost of the migration but didn't reap its benefit. They did the hard work (dual-write, backfill, parallel-run, reconciliation, read-switch) —with all its cost and risk— but by not retiring the old, they don't get the prize that justified that work: a simpler system, with a single store and without the double write cost. They ended up with the complexity of two stores (as before migrating) plus the synchronization machinery (which didn't exist before). It's the worst of both worlds: the complexity of the old state and the intermediate one, at once, forever. Finishing the cut —retiring the old— is the only thing that turns the cost paid into benefit reaped.

Summary and next step

In this lesson you executed the final cut of the data migration: the read-switch and turning off the dual-write. You saw, with the control tower switching from the old radar to the new, that the order is sacred —the controllers look at the new screen first (read-switch), with the old radar still on as backup; only after do they stop feeding it (turn off dual-write); and at the end they dismantle it (retire)—, and that stopping feeding the radar you still watch freezes the image and serves stale data. You executed it step by step: read-switch with the old warm, a write landing in both as backup, turning off the dual-write, and the retirement —and you saw with numbers why the inverted order would serve a stale kbd-mech at 79.99 instead of the real 74.99—. And you kept the closing almost nobody remembers: turning off the dual-write for real and deleting the old, because a migration that ends at the read-switch paid the cost without reaping the prize.

Before moving on you should be able to: order the steps of the cut (read-switch → turn off dual-write → retire) and justify each order; explain why the old stays warm after the read-switch (the rollback window) and why turning off the dual-write burns that net; describe the danger of the inverted order (stale reads from the frozen old); and recognize the migration that doesn't end (eternal dual-write, old not retired) as the silent failure of the cut.

Lesson 8 integrates the whole module into a single executed capstone: migrating the data of Mercado's catalog from old_db to new_db end to end, as a day-by-day migration journal. A DataMigration class that turns on the dual-write, does the backfill (with a real bug that skips the products without stock), runs the parallel-run that catches the gap, reconciles to zero, does the read-switch, turns off the dual-write, and retires the old —the seven days of a migration without downtime, with the system alive at every step—. Deliverable: the plan, the executed code, and the justification of why the incremental and verified migration beat "copy everything in a maintenance window".

Resources

  • Martin Fowler, "ParallelChange" — martinfowler.com/bliki/ParallelChange.html. The read-switch and the retirement of the old are the "migrate" and the "contract" of the reads: moving the readers to the new and removing the old, in that order. The entry that frames the final cut. In English.
  • Sam Newman, Monolith to Microservices (O'Reilly, 2019), ch. 4, "Decomposing the Database" — the cut of reads to the new store, keeping the old as backup during the transition, and retiring it when it's no longer used, within the complete process of separating the monolith's database. In English.
  • Stripe Engineering, "Online migrations at scale" — stripe.com/blog/online-migrations. The production account ends exactly here: moving the reads to the new table, keeping the old for a while as a net, and finally retiring it —with the warning to complete the retirement and not leave the double write on—. In English.
  • Martin Fowler, "StranglerFigApplication" (2004) — martinfowler.com/bliki/StranglerFigApplication.html. The symmetry with module 3: retiring the legacy when the traffic reached 100% is the sibling of retiring the old when the reads moved to the new. The same "don't leave the old on just in case". In English.
← PreviousReconciling the discrepanciesNext →Project: migrate Mercado's catalog data without downtime