Module 5 — Budget and Control: Cost, Time, and Context

7. Real speed versus perceived speed

Next →

Description

By the end of this lesson you will be able to take the data you have been logging since lesson 2 (cost per task) and lesson 6 (stop conditions) and turn it into a single defensible number: total cost until integrated code. You will be able to present that number to your team or to whoever asks you to account for it — a boss, a budget committee, a quarterly review — without falling into the empty promise of the multiplier ("we're 3x more productive with AI"), which sounds good on a slide and collapses under the first uncomfortable question. And you will be able to explain, precisely and without fear of the discomfort, what the public evidence today actually says about coding agents and productivity, and what part of that evidence is still genuinely in dispute.

This matters because at some point someone with decision-making power is going to ask you the direct question: "how much does using agents save us?" Answering with someone else's blog headline — or worse, with a made-up figure because it "feels" like you are going faster — is exactly the mistake Module 1 already taught you to fear: the feeling of speed and the measured speed do not always point in the same direction. You are not going to repeat that lesson here. You are going to use the numbers you already instrumented in this module to answer that question with something that survives scrutiny six months later, when someone asks to see the data again.

Connection to the module: every previous lesson gave you a piece — how much a task costs in dollars and iterations (lesson 2), why long context makes a conversation more expensive (lesson 3), when the expensive model pays for itself (lesson 4), when to cut off before a task eats the afternoon (lesson 6). This lesson does not add a new instrument: it teaches you to add up the ones you already have into a single number, and to communicate it with the same honesty you measured it with.

The cost does not end when the agent finishes writing

A baker takes a cake out of the oven. It looks perfect: even, golden, with a firm surface. He sends it to the display case and marks it done on his order list for the day. Two hours later, the center sinks — it had not finished baking inside — and the customer returns it. The baker has to make another one, from scratch.

How much did that order cost? Not just what the second cake cost, the one that actually made it to the customer's table. The first one also spent flour, eggs, forty minutes of oven time, and the labor of whoever decorated it before sending it to the display case. That spend is real, it happened, and it does not disappear just because the cake that generated it never made it to the table. The order's real cost is the sum of both bakes, even though the baker's order list only shows the second one as "delivered".

A task delegated to a coding agent leaves the same trail when something does not work on the first try. The agent can deliver something that looks finished — the tests pass, the diff looks reasonable — and that result, just like the freshly baked cake, has not yet passed the test that really matters: staying up, integrated, and stable, days later. If that first attempt gets abandoned because it triggered one of the previous lesson's stop conditions — the same fix repeated, a diff that grows without the problem moving — its cost does not disappear from the total just because it was not the attempt that ended up merged. The total cost until integrated code is the sum of all the attempts that led there, not just the last one.

This is not a cosmetic variation on "time to stable merge", the concept you already saw in the previous module of this guide. It is the same principle applied backward in time, not just forward: just as a task's cost keeps running after the merge (while you wait to see whether it comes back as an incident), it also runs before the merge, in any abandoned attempt that consumed real tokens and real minutes of your attention before you decided to change strategy.

Full worked example

Let us build the number with a real task, extending the same logging habit from lesson 2 — a CSV file you have already been filling in — with a column you had not needed yet: which task each attempt belongs to, so you can add them up even when they happened in different sessions, with different models, on different days.

The task: "The report export endpoint fails intermittently under load; there is no visible pattern in the logs."

Attempt 1 — Sonnet 5, same day. After the second fix that attacks the same symptom without resolving it — the previous lesson's unproductive-loop sign — the three-iteration hard limit you defined ahead of time triggers. The decision, following that same lesson's four options, is to switch models. The attempt is abandoned with no integrated code.

Attempt 2 — Opus 4.8, new session. As you saw in lesson 4, escalating mid-conversation invalidates that conversation's cache discount, so it is worth closing the previous session and starting a new one directly with the more capable model. This time the agent finds that the failure only happens when two workers process the same report in parallel and collide on the same temporary file. The fix gets integrated and does not fail again over the following ten days.

Here is what the log looks like, with a shared task_id tying both attempts to the same real task:

task_id,task_type,model,cost_usd,iterations,review_minutes,rework_minutes,status
BUG-207,bug-fix-small,sonnet-5,0.34,2,4,0,integrated
RPT-114,report-export-fix,sonnet-5,0.42,3,14,0,abandoned
RPT-114,report-export-fix,opus-4.8,0.68,2,20,0,integrated

(BUG-207 is a single-attempt task, so you can see the contrast. The numbers are a made-up sample to illustrate the method — yours will look different.)

A short script, extending the one you already used in lesson 2, adds up every attempt by task_id and converts human minutes into dollars using a loaded rate — a hypothesis you declare upfront, not a universal number:

# total_cost_until_integrated.py
# Adds up ALL attempts for a single task -integrated or abandoned-
# and expresses the total cost as a single number, in dollars.
import csv
from collections import defaultdict

HOURLY_RATE_USD = 75.0  # loaded-rate hypothesis for an engineer; adjust to your own team


def load_attempts(path: str) -> list[dict]:
    """Reads the attempt log, one row per attempt, tied together by task_id."""
    with open(path, newline="", encoding="utf-8") as f:
        return list(csv.DictReader(f))


def total_cost_until_integrated(rows: list[dict], hourly_rate: float = HOURLY_RATE_USD) -> dict:
    """Groups by task_id and sums every attempt, regardless of its status."""
    totals = defaultdict(
        lambda: {"cost_usd": 0.0, "human_minutes": 0.0, "task_type": None, "attempts": 0}
    )
    for row in rows:
        task_id = row["task_id"]
        totals[task_id]["cost_usd"] += float(row["cost_usd"])
        totals[task_id]["human_minutes"] += float(row["review_minutes"]) + float(row["rework_minutes"])
        totals[task_id]["task_type"] = row["task_type"]
        totals[task_id]["attempts"] += 1

    summary = {}
    for task_id, data in totals.items():
        human_cost = (data["human_minutes"] / 60) * hourly_rate
        summary[task_id] = {
            "task_type": data["task_type"],
            "attempts": data["attempts"],
            "token_cost_usd": round(data["cost_usd"], 2),
            "human_cost_usd": round(human_cost, 2),
            "total_cost_usd": round(data["cost_usd"] + human_cost, 2),
        }
    return summary


if __name__ == "__main__":
    rows = load_attempts("task-lifecycle.csv")
    summary = total_cost_until_integrated(rows)
    for task_id, stats in sorted(summary.items()):
        print(
            f"{task_id:10s} {stats['task_type']:20s} attempts={stats['attempts']} "
            f"tokens=${stats['token_cost_usd']:.2f} human=${stats['human_cost_usd']:.2f} "
            f"total=${stats['total_cost_usd']:.2f}"
        )

Command:

python total_cost_until_integrated.py

What to expect:

BUG-207    bug-fix-small        attempts=1 tokens=$0.34 human=$5.00 total=$5.34
RPT-114    report-export-fix    attempts=2 tokens=$1.10 human=$42.50 total=$43.60

RPT-114 cost more than eight times what BUG-207 cost, and that difference does not come mainly from tokens — $1.10 versus $0.34, barely three times more. It comes from having spent two full sessions of human attention instead of one, and from the abandoned attempt not stopping being a cost just because it never made it to the customer's table. If you had reported this task's cost using only the session that actually worked — $0.68 in tokens plus that session's 20 review minutes, about $25.68 total — you would have underestimated the real cost by more than 40%. That is exactly the mistake the next section teaches you not to make when you present this number externally.

How to present it without the empty promise of the multiplier

A slide says: "This quarter we adopted coding agents. The team is 3x more productive." It sounds good, it is short, and it is exactly the kind of claim that does not survive the first serious question: 3x from what baseline, measured how? Is that 3x the average of which tasks, and what happened to the ones that did not improve? Can anyone reproduce that calculation with the raw data? If the answer to any of those questions is "I don't know" or "we didn't write it down", the figure is not data: it is an empty promise, and the day someone puts it to the test — a feature that came out more expensive, not cheaper, as you saw can happen in lesson 2 — that figure takes your credibility down with it.

The alternative is not harder to build; it is just more honest, and you already have this module's three ingredients to put it together:

  • Report it by task type, never as a single average. Mixing a mechanical rename with a feature with a weak specification into the same "3x" hides exactly the variance lesson 4 taught you to tell apart. Use the cost-by-task-type table you have already been building.
  • State your assumptions alongside the number. The loaded rate you used to convert minutes into dollars, the stability window you required before counting a task as "integrated", your sample size (n). Without those three pieces of data, nobody else can reproduce your calculation, and a number that cannot be reproduced cannot be defended.
  • Compare against your own baseline, not against an internet average. The same mistake lesson 2 named for comparing costs between tasks applies here for comparing "before" against "after": your baseline is the manual work log from Module 1, not a figure from someone else's case study with different code, a different team, and a different specification.
  • Show the categories that did not improve too. If a task category came out more expensive — not cheaper — once rework is added in, say so. It is exactly the kind of honesty that sets a credible report apart from a marketing promise, and it is easier to defend at the next review than explaining why last time's number did not hold up.

Here is what the difference looks like with real data from the earlier example, plus the lesson 2 log:

Task typenTotal cost before (estimated, Module 1 baseline)Total cost today (measured, with agent)Change
bug-fix-small8~$45$5.34 average-88%
code-review-pass6~$60$12.06 average-80%
feature-new5~$80$35.90 average, with active rework-55%, watch closely
refactor-module4~$70$19.15 average-73%

Loaded rate used: $75/hour. Stability window: two weeks without reversion. The "before" column is a manual estimate from your own Module 1 log, not measured with the same rigor as the "today" column — naming that bias explicitly, instead of hiding it, is exactly what the next lesson's project is going to ask you to do more carefully.

This table does not say "we're 3x faster". It says something more useful and more defensible: in three out of four task types, total cost dropped substantially, and in the fourth it also dropped but with a rework signal worth watching before declaring victory. That sentence, with the table behind it, survives the uncomfortable question. The "3x" alone, without the table, does not.

What the public evidence says, and what is still in dispute

You already saw in module 1 the most cited finding about this gap: the METR study where experienced developers ended up 19% slower with AI while remaining convinced they had been 20% faster. What you did not see there — and it is worth looking at closely before citing that number in a serious conversation — is what the study itself says about its own limits, because that is the key to why the public evidence, taken as a whole, is contradictory.

METR's authors were explicit: their sample was 16 developers working on large, mature open-source repositories — over 22,000 stars and a million lines of code on average —, with real tasks typically taking two hours. They themselves warn that their result does not demonstrate that "current AI systems fail to speed up the majority of developers" in general: it is, in their own words, a snapshot of early-2025 AI's capabilities in one particular setting. They also warn they cannot rule out learning effects beyond the 50 hours of use of the specific tool they studied, and that AI likely performs worse precisely in the kind of environment they chose: code with very high quality standards and many implicit requirements that take a human time to learn — exactly the profile of a large, old open-source repository, not just any programming task.

Contrast that with GitHub's original randomized Copilot study: developers completed a bounded, well-specified task — writing an HTTP server in JavaScript — 55% faster with the tool. That is a real figure, from a real controlled experiment, and it points in exactly the opposite direction from METR. The difference is not that one of the two studies is poorly done: it is that they measure completely different task profiles. An HTTP server built from scratch, with a closed success criterion, is the terrain where an agent performs best, as you already saw in this module's lesson 4. Debugging an intermittent failure in a million lines of someone else's code, with implicit requirements nobody wrote down, is the opposite terrain.

DORA's 2025 report adds one more nuance, different from the 2024 edition you already cited in the previous module: while the 2024 report found a perceived improvement at the individual level alongside a measured drop in team stability, the 2025 edition finds that AI adoption is associated with a measured — not just perceived — improvement in delivery throughput and product performance. The drop in delivery stability, however, remains. And the report itself names what separates the teams that win from the ones that lose: it is not which model they use, it is whether they have decoupled architectures, fast feedback loops, and robust automated tests — the framework the report calls "AI as amplifier": it magnifies whatever strength or weakness the team already had, it does not replace it.

That is where the real dispute is, and it is not "does AI help or not?" — that question no longer makes sense to ask that way. What is in dispute is what the result depends on, and the public evidence points, consistently across studies that appear to contradict each other, to three variables: how closed the task profile is (lesson 4), how mature the specification and verification practices are of the team running it (Modules 2 and 4 of this guide), and which model generation was being used at the time of the study — something that changes faster than any study can be repeated. None of those three factors is the same between your team and METR's 16 open-source repository developers, or between your team and GitHub's experiment participants. That is why nobody can give you, today, a universal number for "how much faster is working with agents": METR's number exists, GitHub's number exists, DORA's exists, and all three are real and, at the same time, incomplete for your case, because none of them measured your task, your team, or your specification discipline.

The honest answer you give your boss when they ask for that universal number is not citing whichever study suits you best. It is telling them that number does not exist in the abstract — yours exists, measured with your own table, like the one you built in the previous section.

Common mistakes

Reporting an aggregate multiplier that mixes very different task types (conceptual). What happens: someone calculates "3x faster" by dividing the total cost of every task in the quarter, without distinguishing a mechanical rename from a feature with a weak specification, and presents that single number as if it described any future task. Why it happens: an aggregate average is easier to cite on a slide than a four-row table, even though the table is what actually describes what happened. How to spot it: if nobody can tell you which task type performed worst inside that average, the number is hiding its own variance. How to fix it: always report by task type, with its own n, like the table in the previous section; the aggregate average, if someone asks for it, is a summary derived from that table, not the main data point.

Resolving the "contradiction" by citing only the study that suits you, instead of explaining what sets them apart (conceptual). What happens: in front of a skeptic, someone cites only GitHub's 55% to justify more investment in agents; in front of a cautious committee, someone cites only METR's 19% to slow it down. In both cases the figure is used as a rhetorical weapon, not as complete evidence. Why it happens: it is faster to cite a number than to explain why two real, well-conducted studies measured different things — task profile, practice maturity, model generation. How to spot it: if the conversation never mentions what type of task or population the cited study measured, the figure is being used without its context. How to fix it: whenever you cite either of these studies, name in the same sentence the task profile or population it measured — "on well-specified tasks like this one, GitHub's evidence..." —, not the figure alone.

Calculating total cost until integrated using only the session that worked, forgetting the abandoned attempts (practical). What happens: when reporting how much a task cost, only the cost of the final session — the one that actually produced integrated code — gets counted, and the earlier attempts that were abandoned after triggering a previous-lesson stop condition get ignored. Why it happens: those attempts left no code in the repository, so they feel like they did not "count", even though they consumed real tokens and real minutes. How to spot it: if your cost-per-task log has no column tying several attempts to the same task_id, you have no way to add up the abandoned ones. How to fix it: use a task identifier shared across attempts, as in the worked example, and add up every attempt — integrated or not — before reporting that task's cost.

Exercises

Exercise 1. A task (AUTH-88, type auth-bug-fix) had two attempts: the first with Sonnet 5 cost $0.51 in tokens, with 10 minutes of review and 0 of rework, and was abandoned when it triggered the previous lesson's iteration limit. The second, with Opus 4.8 in a new session, cost $0.94 in tokens, with 16 minutes of review and 0 of rework, and did get integrated. Calculate this task's total cost until integrated code using a loaded rate of $60 per hour.

See solution

Token cost: $0.51 + $0.94 = $1.45.

Total human minutes: (10 + 0) + (16 + 0) = 26 minutes = 26/60 = 0.4333 hours.

Human cost: 0.4333 × $60 = $26.00.

Total cost until integrated: $1.45 + $26.00 = $27.45.

Why it works: the total cost adds up both attempts — the abandoned one and the integrated one — because both consumed real tokens and attention before reaching the final result. Reporting only the second attempt ($0.94 in tokens plus $16.00 of review, $16.94 total) would have underestimated the real cost by nearly 40%, the same mistake you saw in the worked example with RPT-114.

Exercise 2. A team measures, by task type, the cost change after adopting agents: bug-fix-small -80%, code-review-pass -75%, feature-new +10% (more expensive than before, due to rework). In their quarterly report they write: "We adopted coding agents this quarter. On average, the team is 3x faster." What mistake from this lesson does that claim make, and how would you rewrite it?

See solution

It makes the mistake of reporting an aggregate multiplier that mixes very different task types. The "3x" averages three categories that improved with one that got worse, and it hides exactly the category that most needs attention — feature-new, which came out 10% more expensive, not cheaper, once rework is counted. Anyone who receives that report and later discovers the feature-new detail is going to distrust the rest of the number too, and rightly so.

An honest rewrite: "We adopted coding agents this quarter. On bug-fix-small and code-review-pass, total cost dropped 80% and 75% respectively (n=8 and n=6). Refactor-module dropped similarly. On feature-new, however, total cost rose 10% against our baseline, due to rework following tasks with weak specifications; we are reviewing how to specify that category better before delegating it, following what we saw in the specification module." It is longer, but every sentence survives the question "how did you measure that?"

Why it works: a claim broken down by category, including the ones that did not improve, cannot be discredited by a single case — it already acknowledges upfront that it exists, and explains what is being done about it.

Exercise 3. A colleague tells you: "I read that METR found developers are 19% slower with AI, so it doesn't make sense for us to keep investing in this." In three or four sentences, explain why that study and the 55%-faster finding from the original GitHub Copilot experiment do not actually contradict each other, and what determines which side your own team's work falls on.

See solution

They do not contradict each other because they measured nearly opposite task profiles: GitHub measured a bounded, well-specified task built from scratch (writing an HTTP server), the terrain where an agent performs best; METR measured experienced developers navigating large, mature open-source repositories, with many implicit requirements — the terrain where, according to METR's own authors, AI performs worst. METR's authors even explicitly warn that their result does not show AI fails to speed up most developers; it is a snapshot of one particular setting. What determines which side your own team's work falls on is not which study is "right", but how closed your tasks' profile is and how mature your own specification and verification practices are — the same variables the 2025 DORA report points to as what separates the teams that win with AI from the ones that lose.

Why it works: treating the question as "which study is correct?" ignores that both are correct for what they measured; the useful question is "does my work look more like GitHub's task or METR's?", and only your own table can answer that, not someone else's headline.

Summary and next step

You learned to convert the data you already instrumented — cost per task from lesson 2, stop conditions from lesson 6 — into a single defensible number: total cost until integrated code, which includes the abandoned attempts and not just the one that made it to the table. You learned to present that number by task type, with your assumptions stated and compared against your own baseline, instead of risking your credibility on a single multiplier that does not survive the first uncomfortable question. And you saw that the public evidence — METR, GitHub Copilot's original experiment, DORA 2025 — is not contradictory by anyone's mistake: it measures different task profiles and maturity levels, and that is exactly why there is no universal number you can borrow. Yours exists, and you already know how to calculate it.

Before moving on you should be able to: take the log of a task with several attempts and calculate its total cost until integrated by adding up every attempt, not just the successful one; point out the mistake in a report that averages very different task types into a single multiplier; and explain in three sentences, without citing a single headline, why two public studies that appear to contradict each other are actually measuring different things.

What you did here — turning raw instrumentation into a defensible number — is exactly what this module's closing project is going to ask you to do systematically: five real tasks, with this same criteria, including re-measuring two tasks from Module 1 with the discipline you have now and did not have then — and naming, without hiding it, the bias that comparison carries from its own baseline.

Resources

← Previous6. Stop conditions and spend limitsNext →8. Project: cost-instrumented workflow