Module 7: Measuring Migration Progress
Measure the result, not the effort
Overview
Before building any dashboard, there's a decision that determines everything: what to measure. And it's more treacherous than it seems, because there are two families of metrics that look equally "progress-like" but say opposite things. There are the effort metrics —how many story points the team burned, how many hours it put in, how many tasks it closed, how many PRs it merged— and there are the result metrics —how many calls to the legacy were really cut, what percentage of the traffic already lives on the new route, how many endpoints and tables left the monolith—. This lesson shows, executed, why only the result ones tell the truth about a migration, and why the effort ones almost always run ahead and make you believe you're almost finishing when you're barely a quarter of the way.
The difference isn't philosophical; it's measurable. The effort measures what the team did; the result measures what happened to the legacy. And those two numbers separate, sometimes dramatically, because a huge part of a migration's work —setting up the infrastructure, writing the characterization tests, building the anti-corruption layer, preparing the strangler facade— burns a lot of effort without yet cutting a single call to the legacy. That work is necessary, but it's not result: the legacy keeps serving exactly the same traffic as before. If you measure the effort, your progress bar jumps to 60% while the legacy didn't move a millimeter. If you measure the result, the bar tells the uncomfortable truth: 0%, because nothing has been cut yet.
Connection with the module. This is the lesson that fixes what to measure, and that's why it goes before all the ones that build metrics. Lesson 3 builds the burn-down —a result metric par excellence (legacy_calls)—; lessons 4 and 5 build the fitness function —which also measures a result: whether the legacy grew—; and lesson 6 defines the done over results (legacy_calls == 0, tables cut), never over effort ("we put in 2000 hours"). All the rest of the module rests on this lesson's decision. Notice the boundary: here we don't discuss how to estimate story points or how to plan sprints —that's project management—; we discuss, within a migration, which of the two families of metrics reflects the real progress toward turning off the legacy. The effort has its place (planning, sizing the remaining work); but as a migration progress signal, it deceives.
An analogy: measuring a trip by hours at the wheel or by kilometers left
Imagine you drive from one city to another, 600 kilometers, and you want to know how far you've advanced. You have two ways to measure it. The first: how many hours you've been at the wheel. You've been driving for six hours, you're tired, you made an enormous effort —six hours is a lot—. The second: how many kilometers you have left to arrive. You have 500 left. Did you advance? By the hours, a lot: six hours of hard work. By the kilometers, almost nothing: you covered 100 of 600.
How can that be? Because the hours at the wheel measure your effort, not your advance toward the destination. Maybe you got lost and drove in circles. Maybe there was traffic and you advanced at five kilometers an hour. Maybe you spent two hours looking for parking in a town. All that is hours —real effort, real tiredness— that didn't bring you closer to your destination. The hours rise even if you don't arrive; the kilometers left only drop when you really get closer.
A migration is identical. The story points are the hours at the wheel: they measure how much the team worked, and they rise with each sprint even if the legacy hasn't moved. The calls to the legacy that remain are the kilometers left: they only drop when you really cut a dependency of the monolith. If your progress report speaks of burned points, you're measuring hours at the wheel —you can report "80% of the effort" while the legacy keeps serving 80% of the traffic—. If your report speaks of calls to the legacy cut, you're measuring the kilometers left: the only figure that drops when you really arrive.
Worked example: the same migration measured both ways
We're going to take a single migration —Mercado's catalog— and measure it with both families of metrics at the same time, sprint by sprint. On the effort side: how many story points the team accumulated, out of 100 planned, and its effort_percent. On the result side: how many legacy_calls remain over the fixed sample of 1000 requests, and its outcome_percent (the traffic that no longer touches the legacy). And a third column, the gap: how far ahead the effort is of the result. The data is fixed so the pattern is crisp and reproducible.
# Measure EFFORT vs measure RESULT. The same migration, two metrics.
# The effort (story points burned) runs ahead and deceives; the result
# (calls to the legacy that were really cut) tells the truth.
SAMPLE = 1000 # catalog requests measured per sprint
POINTS_TOTAL = 100 # story points planned for the migration
# Per sprint: (accumulated points, calls to the legacy remaining)
sprints = [
(10, 1000),
(35, 980),
(60, 900),
(80, 760),
(92, 400),
(100, 0),
]
def effort_percent(points_done):
return points_done / POINTS_TOTAL * 100
def outcome_percent(legacy_calls):
# the real result: what fraction of the traffic NO longer touches the legacy
return (SAMPLE - legacy_calls) / SAMPLE * 100
print("Effort vs result: two ways to measure the SAME migration\n")
print(f"{'sprint':>7}{'points':>8}{'effort%':>9}{'legacy_calls':>14}"
f"{'outcome%':>10}{'gap':>9}")
print("-" * 57)
for i, (points_done, legacy_calls) in enumerate(sprints):
ep = effort_percent(points_done)
op = outcome_percent(legacy_calls)
gap = ep - op
print(f"{i:>7}{points_done:>8}{ep:>8.0f}%{legacy_calls:>14}"
f"{op:>9.0f}%{gap:>8.0f}")
print("-" * 57)
print("\n Sprint 3: the effort says 80% done; the result says 24%.")
print(" Whoever reports effort% claims victory; outcome% (legacy_calls -> 0)")
print(" is the only one that doesn't lie. Only when both reach 100 is it over.")What to expect. When you run the file, the output is exactly this:
Effort vs result: two ways to measure the SAME migration
sprint points effort% legacy_calls outcome% gap
---------------------------------------------------------
0 10 10% 1000 0% 10
1 35 35% 980 2% 33
2 60 60% 900 10% 50
3 80 80% 760 24% 56
4 92 92% 400 60% 32
5 100 100% 0 100% 0
---------------------------------------------------------
Sprint 3: the effort says 80% done; the result says 24%.
Whoever reports effort% claims victory; outcome% (legacy_calls -> 0)
is the only one that doesn't lie. Only when both reach 100 is it over.
Read the two percentage columns in parallel, row by row, and look at the gap between them.
In sprint 0, the team already burned 10 points (10% of the effort) but the outcome_percent is 0%: the 1000 calls are still in the legacy. That 10% of effort went into setting up the migration's infrastructure —the strangler facade, the instrumentation— without cutting anything yet. The gap is already 10.
In sprints 1 and 2, the gap widens: the effort is at 35% and 60%, but the result is barely at 2% and 10%. The team worked a lot —wrote characterization tests, built the anti-corruption layer, prepared the data migration—, all real and necessary work, but the legacy keeps serving 980 and 900 of the 1000 requests. If someone reported only the effort_percent, they'd say "we're at 60%, more than half"; the outcome_percent tells the truth: 10%, barely starting to cut.
Sprint 3 is the moment of maximum lie: effort_percent = 80%, outcome_percent = 24%, gap of 56. Eighty percent of the effort burned, and only a quarter of the traffic cut from the legacy. Imagine the report to management: "we're at 80%". Everyone understands "we're almost done, one more push". But the legacy still serves 760 of the 1000 requests —76% of the traffic is still in the system you're supposedly turning off—. The effort bar is at 80%; the real migration is at 24%. Whoever trusts the effort is going to promise an end that won't arrive when they promised it.
In sprints 4 and 5, the result finally accelerates and catches up to the effort: the gap closes from 56 to 32 to 0. This is where the infrastructure work of the first sprints "gets paid": with the tests, the ACL, and the strangler already built, cutting calls to the legacy becomes fast, and the outcome_percent rises from 24% to 60% to 100%. In the end, both reach 100 —and only then did the migration really finish—.
The shape of the gap tells the story: the effort rises steadily and early; the result lags at first (all the preparation work cuts nothing) and then accelerates. If you measure the effort, your bar lies at the worst moment —halfway through the project, right when someone asks "when do you finish?"—. If you measure the result, your bar is slower and more honest: it never tells you "almost" until it really is almost.
Deep dive: why the effort always runs ahead
The gap of the example isn't coincidence or bad luck: it's the structural shape of almost every migration. And understanding why vaccinates you against believing the effort.
The reason is that the work of an incremental migration is front-loaded. Before cutting the first call to the legacy, you have to: understand the legacy and characterize it (M2), build the facade (M3), build the new service (M3, M5), set up the anti-corruption layer (M5), prepare the data migration (M6), and instrument everything to measure. That's a mountain of effort —it can be 40% or 50% of the total work— that produces zero result in terms of legacy_calls: the legacy, when all that preparation is done, serves exactly the same traffic it served on day one. The effort shot up; the result didn't move.
100 ┤ effort ● ● ● ● ● ●
│ ● ● ●
% │ ● ●
│ ● outcome ○
│ ● ○ ○
│ ● ○
│● ○
0 ┤● ○ ○ ○ ○ ○ ○
└──────────────────────────────────────────────────
sprint 0 3 (max gap) 5 (closes)
The effort (the filled dots) rises in an almost straight line from sprint 0. The result (the empty circles) stays glued to the floor while the infrastructure is being built, and only takes off when that infrastructure already exists and allows cutting calls fast. The maximum gap occurs halfway —right where the effort is already very high and the result is barely starting—. That midpoint is where promises die: the team feels it's at 80% (from the effort invested) and promises a near end, but the result says 76% is left.
There's an important practical consequence: the effort isn't good for projecting the end date, but the result is. If you extrapolate the effort line, you predict a false (early) end. If you extrapolate the slope of the result —the burn-down of legacy_calls, which is lesson 3—, you predict the real end. That's why the burn-down, and not the "points burned", is this module's projection instrument.
This does not mean measuring the effort is wrong in general. The effort is useful for planning (how much total work is there? do we have capacity?) and for sizing what's left. What the effort shouldn't do is pass itself off as migration progress. They're two different questions: "how much has the team worked?" (effort) and "how much has the legacy died?" (result). The second is the one the business cares about, the one that decides when you can turn off the old system, and the only one that honestly reaches 100%.
A quick test to know if a metric is result or effort: ask yourself "if the team worked the whole sprint but didn't cut a single call to the legacy, would this metric rise?". If the answer is yes (the story points rise, the hours rise, the PRs rise), it's an effort metric and doesn't measure the migration's progress. If the answer is no (the legacy_calls only drop if you really cut a dependency), it's a result metric. Measure with the ones that answer "no".
Common mistakes
Reporting the migration's progress in story points. What happens: the progress report says "we burned 80 of 100 points, we're at 80%", and everyone —including management— understands the migration is almost finished. Why it happens: the story points are already measured for the sprint, they're at hand, and "80 of 100" sounds like clear progress. How to spot it: the progress percentage the team reports comes from points, hours, or tasks, not from a measurement of the legacy; nobody can answer "what fraction of the traffic no longer touches the legacy?". How to fix it: report the result —the outcome_percent, the legacy_calls remaining, the cut endpoints—. The points measure how much the team worked, not how much the legacy died, and the two things separate right at the middle of the project (gap of 56 in the example). An honest migration report speaks of what happened to the legacy, not of what the team did.
Confusing "we finished the infrastructure" with "we're halfway". What happens: the team finishes the facade, the ACL, and the tests —an enormous amount of work— and concludes "we already did the hardest half, the rest is downhill, we're at 50%". Why it happens: the infrastructure is a lot of work and is the hardest part conceptually, so "finishing it" feels like a midpoint milestone. How to spot it: the outcome_percent (calls to the legacy cut) is still near 0 even though the team declares "50% done"; the celebrated milestone is effort, not result. How to fix it: finishing the infrastructure is a real milestone, but of effort, not of result —the legacy didn't move—. It's true that afterward the result accelerates (sprints 4 and 5 of the example), but as long as legacy_calls doesn't drop, the progress of the migration is what the result says, not what you feel from the work done. Celebrate the infrastructure for what it is (a capability built), and measure the advance with the legacy dying.
Not instrumenting the system to measure the result. What happens: the team would like to measure the result, but has no way to count how many calls go to the legacy and how many to the new, so by default it reports the only thing it does measure: the effort. Why it happens: measuring legacy_calls requires instrumenting the strangler router or the code to count where each request goes, and that's work that's sometimes skipped. How to spot it: when you ask for the outcome_percent, the answer is "we don't have it measured"; the only available datum is points or hours. How to fix it: instrument from day one. The strangler router (M3) already knows where it sends each request —you just have to count it—; the burn-down (lesson 3) feeds on that count. Without instrumentation there's no result metric, and without a result metric the migration flies blind guided by the effort. Measuring the result isn't optional: it's what makes all the other lessons of this module possible.
Exercises
Exercise 1 — Classify the metrics. For each one, say whether it's effort or result, and apply the quick test ("would it rise if the team works but cuts no call to the legacy?"): (a) story points burned in the sprint; (b) percentage of the traffic served by the new route; (c) number of PRs merged; (d) number of tables the monolith no longer shares; (e) hours invested by the team.
See solution
- (a) Story points burned → effort. Test: if the team works a whole sprint refactoring without cutting a call to the legacy, the points rise anyway. Rises without result → effort.
- (b) % of the traffic on the new route → result. Test: it only rises if you really diverted traffic from the legacy to the new. It doesn't rise from working; it rises from cutting. → result.
- (c) PRs merged → effort. Test: you can merge ten preparation PRs (tests, infrastructure) without moving a single call to the legacy. Rises without result → effort.
- (d) Tables the monolith no longer shares → result. Test: it only drops when you really cut the dependency of a table. It's a real cut of the legacy → result.
- (e) Hours invested → effort. Test: the hours rise while the team is working, even if the legacy doesn't move. The purest case of effort → effort.
The rule holds: the result ones (b, d) point at the legacy shrinking; the effort ones (a, c, e) point at the team's activity. Only the former measure the migration's progress.
Exercise 2 — Explain the sprint 3 gap. In the output, sprint 3 had effort_percent = 80%, outcome_percent = 24%, gap of 56. (a) Where did that 80% of effort go if only 24% of the traffic was cut? (b) If you report "80% done" to management, what will they expect, and why will you be wrong? (c) Why does the gap start to close from sprint 4?
See solution
(a) That 80% of effort went, for the most part, into building the migration's infrastructure that doesn't yet cut calls: characterizing the legacy, setting up the strangler facade, building the anti-corruption layer, preparing the data migration, instrumenting. All that work is real and necessary, but it doesn't reduce legacy_calls —the legacy, after doing it, keeps serving 760 of 1000—. The effort was front-loaded; the result is barely starting.
(b) If you report "80% done", management is going to expect the migration to finish soon —"20% left, a couple of sprints"—. You'll be wrong because the 80% is effort, not result: the legacy still serves 76% of the traffic, and turning it off requires cutting that 76%, which is most of the result work that's left. You promised a near end measuring the metric that runs ahead; the real end is marked by the result, which says 24%.
(c) Because from sprint 4 the infrastructure is already built, and cutting calls to the legacy becomes fast: with the facade, the ACL, and the tests ready, each sprint can divert much more traffic. The outcome_percent accelerates from 24% to 60% to 100%, catching up to the effort. The gap closes because the result finally "cashes in" the infrastructure investment of the first sprints. This also explains why the effort isn't good for projecting: its slope doesn't anticipate this change of pace of the result.
Exercise 3 — Design the report. You're asked for a one-line weekly report on the progress of the catalog's migration, for management. (a) Write a bad version (effort-based) and a good one (result-based) for sprint 3. (b) What result metric would you include besides the outcome_percent? (c) How would you keep "we're at 24%" from sounding like the team didn't work, when in reality it burned 80% of the effort?
See solution
(a) Bad (effort): "Catalog migration at 80%: we burned 80 of 100 points, we're almost done." It sounds like a near end, and it's false. Good (result): "The catalog's 24% of traffic already lives on the new service (760 of 1000 requests still touch the legacy). The infrastructure is ready, so the cutting pace will accelerate in the coming sprints." It tells the truth of the legacy and gives context on the pace.
(b) Besides the outcome_percent, I'd include the burn-down of legacy_calls with its trend (how many were cut this week and the projection of when it reaches zero, which is lesson 3), and the count of endpoints and tables cut from the monolith. Those result metrics, together, paint the real advance and allow projecting the end —something the effort can't do—.
(c) By explicitly separating the two questions in the report: "The team burned 80% of the planned effort (the infrastructure is ready, which was the hardest part); and as a result, 24% of the traffic no longer touches the legacy, with the pace accelerating now that the infrastructure exists." This way you acknowledge the work done (effort) without letting it disguise itself as migration progress (result). The trick is not to hide the effort, but to not confuse it with the result: they're two true and distinct things, and the honest report shows both labeled for what they are.
Summary and next step
In this lesson you fixed the decision that holds up the whole module: measure the result, not the effort. You saw, with the trip measured by hours at the wheel (effort) versus kilometers left (result), that the two families of metrics look equally "progress-like" but say opposite things —one rises from working, the other only drops from cutting the legacy—. And you executed it over the same migration: the effort_percent running ahead (80% in sprint 3) while the outcome_percent told the truth (24%), with a maximum gap of 56 right halfway that then closes. You learned why the effort always runs ahead (the work is front-loaded and produces no result until the infrastructure exists), and the quick test to distinguish the two: "would this metric rise if the team works but cuts no call to the legacy?".
Before moving on you should be able to: classify a metric as effort or result with the quick test; explain why the effort runs ahead and gives a false sense of the end; read a gap between effort_percent and outcome_percent and say what it means; and write an honest progress report that doesn't disguise effort as progress.
Lesson 3 takes the result metric par excellence —the legacy_calls— and builds with it the module's central instrument: the burn-down. You're going to see the descending slope of the calls to the legacy, sprint by sprint, to zero, and you'll learn to read what really matters: not the value of an isolated sprint, but the trend —the velocity with which they drop— and the projection derived from it. It's the lesson that turns "24% of the traffic is migrated" into "at this pace, we'll finish in sprint such", which is what whoever awaits the end really wants to know.
Resources
- Sam Newman, Monolith to Microservices (O'Reilly, 2019), ch. 3 — Newman insists on measuring a migration's progress by what happens to the monolith (functionality retired, calls cut), not by the team's activity. The basis of this lesson's distinction. In English.
- Martin Fowler, "StranglerFigApplication" (2004) — martinfowler.com/bliki/StranglerFigApplication.html. The pattern whose goal is for the legacy to die: the only progress that counts is the one that reduces what the legacy serves, not the effort invested in building the new. In English.
- Neal Ford, Rebecca Parsons, and Patrick Kua, Building Evolutionary Architectures (O'Reilly, 2nd ed., 2022) — the book that advocates measuring objective and verifiable properties of the architecture (fitness functions) instead of perceptions; the same spirit of preferring the measurable result to the perceived effort. In English.
- Martin Fowler, martinfowler.com — the bliki with the entries on metrics, evolutionary architecture, and strangler fig that frame why the result is measured and not the activity. In English.