Measuring the Coordination Cost of the Complete System

Description

Four counting lessons, four different formulas: M1 L05 measured a task resolved by a single agent against a supervisor with a single specialist (3 vs. 5 calls, 0 vs. 2 hops). M3 L05 measured a three-stage pipeline against a supervisor deciding at each one (7 vs. 10 calls, 2 vs. 6 hops). M4 L07 measured a sequential fan-out against a parallel one (6 = 6 calls, 4 = 4 hops, but 6 vs. 4 rounds). M5 L07 measured a direct handoff against a return to the supervisor (4 vs. 6 calls, 1 vs. 3 hops). Each one measured its own pattern, in isolation.

This lesson builds the missing piece: coordination_cost, a function that generalizes the three conventions —N - 1 for a pipeline (M3), HOPS_PER_SPECIALIST = 2 for a track the supervisor consults (M4), a handoff's direct hop (M5)— into a single formula that works over any PLAN of tracks, regardless of how many or which pattern. It's this whole module's only genuinely new piece of code, and you're going to test it against the already-built Demos A and B —confirming, with an honest comparison, how it's similar to and how it differs from M3 and M4's isolated measurements.

Connection to the module

This lesson reuses, unchanged, count_model_calls and count_tool_calls from M1 L05, and the hop conventions from M2 L06/M4 L07 (HOPS_PER_SPECIALIST = 2) and M5 L07 (a handoff's direct hop). The only new thing is track_calls_and_hops and coordination_cost, which apply those already-known conventions over a PLAN with several tracks at once — the count Module 1 promised to measure for a complete system, and that M7 L01 explicitly left pending until here. Lesson 08 reuses this exact function, unchanged, over Demo C.

Analogy: the ledger, not just the kitchen

This module's lessons 03, 04, 05, and 06 showed the kitchen works —each mechanism resolves its part—. But a restaurant owner who wants to sell the business doesn't just deliver a kitchen that works: they deliver a ledger that says, with numbers, how much it costs to serve each type of table. This lesson is that ledger, generalized: not a different row for each type of table (the way M3, M4, and M5 each did, with their own spreadsheet), but a single formula that works no matter how many dishes —or patterns— the table you're billing has.

Worked example: coordination_cost, generalizing M3, M4, and M5

import ast
import concurrent.futures
from dataclasses import dataclass
import reservo_tools as rt

# ---- SPECIALISTS, run_specialist, run_pipeline, run_tracks_parallel,
# Track: identical to Modules 2, 3, 4/7 -- the same functions you already
# tested in this module's lessons 02, 03, and 04 ----


def dispatch_parallel(tool_use_blocks, tools):
    with concurrent.futures.ThreadPoolExecutor(max_workers=len(tool_use_blocks)) as pool:
        futures = [pool.submit(tools[b["name"]], **b["input"]) for b in tool_use_blocks]
        results = [f.result() for f in futures]
    return [
        {"type": "tool_result", "tool_use_id": b["id"], "content": str(r)}
        for b, r in zip(tool_use_blocks, results)
    ]


def run_agent_parallel(question, model_script, tools, max_iterations=10):
    messages = [{"role": "user", "content": question}]
    for step in range(max_iterations):
        turn = model_script[step]
        messages.append({"role": "assistant", "content": turn["content"]})
        if turn["stop_reason"] != "tool_use":
            return turn, messages
        tool_result_blocks = dispatch_parallel(turn["content"], tools)
        messages.append({"role": "user", "content": tool_result_blocks})
    raise RuntimeError(f"max_iterations reached ({max_iterations})")


def search_docs(query):
    q = query.lower()
    if "cancel" in q:
        return "[cancellation-policy] You can cancel at no charge up until 2 hours before the reserved time."
    if "no" in q and ("present" in q or "show" in q or "arrive" in q):
        return ("[no-show-policy] If you don't show up for a confirmed booking and don't "
                 "cancel at least 2 hours in advance, Reservo charges 50% of the quoted "
                 "price as a no-show fee.")
    return "No relevant policy was found for that question."


SPECIALISTS = {
    "booking_agent": {"tools": {"get_quote": rt.get_quote, "book_room": rt.book_room}},
    "policy_agent": {"tools": {"search_docs": search_docs}},
    "pricing_agent": {"tools": {"get_quote": rt.get_quote}},
}


def run_specialist(name, task, model_script):
    tools = SPECIALISTS[name]["tools"]
    return run_agent_parallel(task, model_script, tools)


@dataclass
class PipelineStage:
    kind: str
    name: str
    label: str


def last_tool_result(history):
    for m in reversed(history):
        if isinstance(m["content"], list):
            for b in m["content"]:
                if b["type"] == "tool_result":
                    try:
                        return ast.literal_eval(b["content"])
                    except (ValueError, SyntaxError):
                        return b["content"]
    return None


def build_stage_task(stage, payload):
    if stage.kind == "quote":
        return f"Quote {payload['room']} {payload['tier']} {payload['hours']}h."
    if stage.kind == "validate_policy":
        return (f"What's the cancellation policy for a {payload['hours']}h booking "
                 f"of {payload['room']}, before confirming it?")
    if stage.kind == "confirm":
        if payload.get("cleared_to_book"):
            return (f"Book {payload['room']} {payload['tier']} {payload['hours']}h "
                     f"for {payload['member']} -- the cancellation policy was already validated.")
        return (f"Book {payload['room']} {payload['tier']} {payload['hours']}h "
                 f"for {payload['member']}.")
    raise ValueError(f"don't know how to build the task for stage {stage.kind!r}")


def extract_payload(stage, history, payload):
    new_payload = dict(payload)
    result = last_tool_result(history)
    if stage.kind == "quote":
        new_payload["price_cents"] = result["price_cents"]
    elif stage.kind == "validate_policy":
        new_payload["cancellation_policy"] = result
        new_payload["cleared_to_book"] = True
    elif stage.kind == "confirm":
        new_payload["booking_id"] = result["booking_id"]
        new_payload["confirmed"] = result["confirmed"]
    return new_payload


def run_pipeline(stages, model_scripts, initial_payload):
    payload = dict(initial_payload)
    trace = []
    for i, stage in enumerate(stages):
        task = build_stage_task(stage, payload)
        final, history = run_specialist(stage.name, task, model_scripts[i])
        payload = extract_payload(stage, history, payload)
        trace.append({"stage": i + 1, "label": stage.label, "history": history})
    return payload, trace


def run_tracks_parallel(jobs):
    results = {}
    with concurrent.futures.ThreadPoolExecutor(max_workers=len(jobs)) as pool:
        future_to_key = {pool.submit(fn): key for key, fn in jobs.items()}
        for future in concurrent.futures.as_completed(future_to_key):
            key = future_to_key[future]
            results[key] = future.result()
    return results


@dataclass
class Track:
    key: str
    pattern: str  # "pipeline" | "fanout" | "handoff"
    description: str


def count_model_calls(history):
    """M1 L05, unchanged: every 'assistant' turn is a model call consumed
    from the (concept) script -- asking for a tool, or the final text."""
    return sum(1 for m in history if m["role"] == "assistant")


def count_tool_calls(history):
    """M1 L05, unchanged: counts the real tool_use blocks dispatched."""
    total = 0
    for m in history:
        if isinstance(m["content"], list):
            total += sum(1 for b in m["content"] if b["type"] == "tool_use")
    return total


PLAN_CALLS = 1     # concept: the supervisor reads the request and builds the track PLAN
COMPOSE_CALLS = 1  # concept: the final synthesis for the member, combining all the tracks
DISPATCH_HOPS = 2  # M2 L06 / M4 L07 convention: a round trip from the supervisor to EACH track


def track_calls_and_hops(track, result):
    """Generalizes, per pattern, how a track's INTERNAL calls and hops
    get counted -- reusing exactly the formulas already measured in M3
    (pipeline_hops = N - 1), M4 (HOPS_PER_SPECIALIST = 2), and M5 (a
    handoff's direct hop = 1)."""
    if track.pattern == "pipeline":
        _, trace = result
        calls = sum(count_model_calls(s["history"]) for s in trace)
        tool_calls = sum(count_tool_calls(s["history"]) for s in trace)
        hops = DISPATCH_HOPS + (len(trace) - 1)
    elif track.pattern == "fanout":
        _, history = result
        calls = count_model_calls(history)
        tool_calls = count_tool_calls(history)
        hops = DISPATCH_HOPS
    elif track.pattern == "handoff":
        _, trace = result
        calls = sum(count_model_calls(step["history"]) for step in trace)
        tool_calls = sum(count_tool_calls(step["history"]) for step in trace)
        hops = DISPATCH_HOPS + 1
    else:
        raise ValueError(f"unknown pattern: {track.pattern!r}")
    return calls, tool_calls, hops


def coordination_cost(plan, track_results):
    """The count Module 1 promised, generalized to a request with N
    tracks of different patterns. Returns model calls, tool calls, hops
    between agents, and coordination rounds (sequential vs. parallel --
    max() instead of sum(), M4 L07's formula, because the tracks run at
    the same time with run_tracks_parallel)."""
    per_track = {}
    for t in plan:
        calls, tool_calls, hops = track_calls_and_hops(t, track_results[t.key])
        per_track[t.key] = {"pattern": t.pattern, "calls": calls, "tools": tool_calls, "hops": hops}

    call_counts = [v["calls"] for v in per_track.values()]
    total_specialist_calls = sum(call_counts)
    total_tool_calls = sum(v["tools"] for v in per_track.values())
    total_hops = sum(v["hops"] for v in per_track.values())
    total_model_calls = PLAN_CALLS + total_specialist_calls + COMPOSE_CALLS

    surrounding = PLAN_CALLS + COMPOSE_CALLS
    sequential_rounds = sum(call_counts) + surrounding
    parallel_rounds = (max(call_counts) if call_counts else 0) + surrounding

    return {
        "per_track": per_track, "total_model_calls": total_model_calls,
        "total_tool_calls": total_tool_calls, "total_hops": total_hops,
        "sequential_rounds": sequential_rounds, "parallel_rounds": parallel_rounds,
    }


# --- Rebuild Demo A's jobs (Luis, lesson 03) and Demo B's (Marta,
# lesson 04) -- the same scripts, with no change at all ---
PIPELINE_STAGES = [
    PipelineStage(kind="quote", name="booking_agent", label="quote"),
    PipelineStage(kind="validate_policy", name="policy_agent", label="validate the cancellation policy"),
    PipelineStage(kind="confirm", name="booking_agent", label="confirm the booking"),
]
INITIAL_PAYLOAD_A = {"room": "Studio", "tier": "pro", "hours": 3, "member": "Luis"}
script_quote_a = [
    {"stop_reason": "tool_use", "content": [{"type": "tool_use", "id": "toolu_01", "name": "get_quote",
     "input": {"room": "Studio", "tier": "pro", "hours": 3}}]},
    {"stop_reason": "end_turn", "content": [{"type": "text", "text": "Studio pro 3h costs 9600 cents."}]},
]
script_policy_a = [
    {"stop_reason": "tool_use", "content": [{"type": "tool_use", "id": "toolu_01", "name": "search_docs",
     "input": {"query": "cancellation policy"}}]},
    {"stop_reason": "end_turn", "content": [{"type": "text", "text": (
        "You can cancel at no charge up until 2 hours before the reserved time. "
        "There's nothing stopping this from being confirmed.")}]},
]
script_confirm_a = [
    {"stop_reason": "tool_use", "content": [{"type": "tool_use", "id": "toolu_01", "name": "book_room",
     "input": {"room": "Studio", "tier": "pro", "hours": 3, "member": "Luis"}}]},
    {"stop_reason": "end_turn", "content": [{"type": "text", "text": "I booked Studio pro 3h for Luis (confirmation #1)."}]},
]


def job_book_studio():
    return run_pipeline(PIPELINE_STAGES, [script_quote_a, script_policy_a, script_confirm_a], INITIAL_PAYLOAD_A)


script_pricing_b = [
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "get_quote", "input": {"room": "Studio", "tier": "pro", "hours": 2}},
        {"type": "tool_use", "id": "toolu_02", "name": "get_quote", "input": {"room": "Boardroom", "tier": "pro", "hours": 2}},
    ]},
    {"stop_reason": "end_turn", "content": [{"type": "text", "text": (
        "Studio pro 2h: 6400 cents. Boardroom pro 2h: 12800 cents. Studio is the cheaper option of the two.")}]},
]
script_no_show_b = [
    {"stop_reason": "tool_use", "content": [{"type": "tool_use", "id": "toolu_01", "name": "search_docs",
     "input": {"query": "what happens if a member doesn't show up for a booking"}}]},
    {"stop_reason": "end_turn", "content": [{"type": "text", "text": (
        "If you don't show up for a confirmed booking and don't cancel at least 2 hours in advance, "
        "Reservo charges 50% of the quoted price as a no-show fee.")}]},
]


def job_compare_rooms():
    return run_specialist("pricing_agent", "Compare Studio and Boardroom pro 2h.", script_pricing_b)


def job_no_show_policy():
    return run_specialist("policy_agent", "What happens if a member doesn't show up for a confirmed booking?", script_no_show_b)


# --- Test the function against Demo A (Luis, 1 track, pipeline) ---
PLAN_A = [Track(key="book_studio", pattern="pipeline", description="quote, validate, confirm")]
results_a = run_tracks_parallel({"book_studio": job_book_studio})
cost_a = coordination_cost(PLAN_A, results_a)

print("=== coordination cost: Demo A (Luis, 1 track, pipeline) ===")
for t in PLAN_A:
    d = cost_a["per_track"][t.key]
    print(f"  {t.key:<14} pattern={d['pattern']:<9} calls={d['calls']} tools={d['tools']} hops={d['hops']}")
print(f"  TOTAL model calls: {cost_a['total_model_calls']}  "
      f"(1 PLAN + {cost_a['per_track']['book_studio']['calls']} specialists + 1 synthesis)")
print(f"  TOTAL hops: {cost_a['total_hops']}")
print(f"  rounds sequential={cost_a['sequential_rounds']} vs. parallel={cost_a['parallel_rounds']}")

# --- Test the function against Demo B (Marta, 2 tracks, fan-out) ---
PLAN_B = [
    Track(key="compare_rooms", pattern="fanout", description="compare Studio and Boardroom pro 2h"),
    Track(key="no_show_policy", pattern="fanout", description="general no-show question"),
]
results_b = run_tracks_parallel({"compare_rooms": job_compare_rooms, "no_show_policy": job_no_show_policy})
cost_b = coordination_cost(PLAN_B, results_b)

print()
print("=== coordination cost: Demo B (Marta, 2 tracks, fan-out) ===")
for t in PLAN_B:
    d = cost_b["per_track"][t.key]
    print(f"  {t.key:<14} pattern={d['pattern']:<9} calls={d['calls']} tools={d['tools']} hops={d['hops']}")
print(f"  TOTAL model calls: {cost_b['total_model_calls']}")
print(f"  TOTAL hops: {cost_b['total_hops']}")
rounds_saved = cost_b['sequential_rounds'] - cost_b['parallel_rounds']
pct = round(100 * rounds_saved / cost_b['sequential_rounds'])
print(f"  rounds sequential={cost_b['sequential_rounds']} vs. parallel={cost_b['parallel_rounds']} "
      f"({rounds_saved} fewer, {pct}% less)")

What to expect:

=== coordination cost: Demo A (Luis, 1 track, pipeline) ===
  book_studio    pattern=pipeline  calls=6 tools=3 hops=4
  TOTAL model calls: 8  (1 PLAN + 6 specialists + 1 synthesis)
  TOTAL hops: 4
  rounds sequential=8 vs. parallel=8

=== coordination cost: Demo B (Marta, 2 tracks, fan-out) ===
  compare_rooms  pattern=fanout    calls=2 tools=2 hops=2
  no_show_policy pattern=fanout    calls=2 tools=1 hops=2
  TOTAL model calls: 6
  TOTAL hops: 4
  rounds sequential=6 vs. parallel=4 (2 fewer, 33% less)

Two results worth reading carefully. Demo A, with a single track, saves nothing in rounds (8 = 8) — that makes sense: max() and sum() over a one-element list give the same number, so running a single track "in parallel" has no other track to share coordination time with. Demo B, with two tracks, does save 2 rounds (33% less) — the same 33% you already saw in M4 L07's canonical example, because the structure is identical: two tracks, two internal calls each.

An honest comparison: this function does NOT reduce exactly to M3 L05

It's worth being precise here, because it would be easy (and misleading) to claim coordination_cost "reproduces" M3 L05's numbers when you pass it a single pipeline track. It doesn't, and there's a real reason:

m3_pipeline_total = 0 + cost_a["per_track"]["book_studio"]["calls"] + 1  # 0 routing (M3) + specialists + 1 synthesis
m3_pipeline_hops = 3 - 1  # N - 1, M3 L05's original formula

print(f"M3 isolated:  {m3_pipeline_total} calls, {m3_pipeline_hops} hops "
      f"(no PLAN_CALLS -- the pattern was already known in advance)")
print(f"M8 system:  {cost_a['total_model_calls']} calls, {cost_a['total_hops']} hops "
      f"(+1 PLAN_CALLS, +2 DISPATCH_HOPS: the supervisor DID have to decide the plan)")

M3 isolated:  7 calls, 2 hops (no PLAN_CALLS -- the pattern was already known in advance)
M8 system:  8 calls, 4 hops (+1 PLAN_CALLS, +2 DISPATCH_HOPS: the supervisor DID have to decide the plan)

The difference is real, not a counting mistake: M3's lesson measured an isolated pipeline, where the pattern was already known in advance —nobody had to decide "is this a pipeline or something else?"—. This module's complete system is different: it receives an arbitrary request and has to decide, first, how many tracks it has and which pattern resolves each one —that's exactly PLAN_CALLS—, and it has to dispatch and get back each track, even if there's just one —that's DISPATCH_HOPS—. That "deciding the plan" cost didn't exist in M3's isolated lesson because, in that lesson, the plan was already given. It's the same kind of finding you already saw in M1 L05 (coordinating costs, even when the real work is identical) — now applied to the cost of deciding which mechanism to use, not just of executing it.

Where each piece of the formula comes from

PLAN_CALLS = 1        -- the supervisor ALWAYS builds a PLAN, regardless
                          of how many tracks result (0, 1, 2, or more).
COMPOSE_CALLS = 1      -- the final synthesis ALWAYS combines what the
                          tracks produced, regardless of how many there are.
DISPATCH_HOPS = 2      -- EVERY track, regardless of its internal pattern,
                          gets dispatched by the supervisor and hands back
                          a result -- a round trip, M4 L07's convention.

INTERNAL hops per pattern (inside DISPATCH_HOPS, not instead of it):
  pipeline (N stages)  -> N - 1   (direct stage-to-stage handoff, M3 L05)
  fanout (1 specialist) -> 0      (nothing to hand off internally)
  handoff              -> 1       (the direct sender -> receiver handoff, M5 L07)

With this table, verifying book_studio's hops=4 in Demo A is straightforward: DISPATCH_HOPS(2) + (3 stages - 1) = 2 + 2 = 4. And verifying each of Demo B's tracks' hops=2: DISPATCH_HOPS(2) + 0 = 2, per track — 2 + 2 = 4 total for Marta's two tracks.

Common mistakes

1. Expecting coordination_cost to give exactly the same numbers as isolated M3/M4/M5. It doesn't, on purpose — the previous section of this lesson explains why: the complete system pays a real decision cost (PLAN_CALLS) an isolated pattern, with the mechanism already decided in advance, didn't pay.

2. Forgetting to add DISPATCH_HOPS to each track's internal hops. A common mistake is calculating only N - 1 for a pipeline and forgetting the supervisor's 2 dispatch/return hops — the result would come out 2 instead of 4 for Demo A, a number that doesn't match the real, executed output.

3. Applying max() instead of sum() for sequential rounds, or vice versa. M4 L07's convention is clear: sequential adds up (sum(), each track waits for the previous one), parallel bounds it to the longest one (max(), everyone runs at the same time). Swapping them would produce a negative round savings, an immediate sign something's wrong.

Exercises

Exercise 1: Calculate the cost of a 3-track PLAN by hand, before running it (Easy)

Without running code yet, calculate total_model_calls and total_hops by hand for a PLAN with three tracks: one 4-stage pipeline, one fanout, and one handoff, where each one's internal calls are 8, 2, and 3 respectively. Then, verify your calculation with code.

per_track_hardcoded = {
    "pipeline_track": {"calls": 8, "hops": 2 + (4 - 1)},
    "fanout_track": {"calls": 2, "hops": 2 + 0},
    "handoff_track": {"calls": 3, "hops": 2 + 1},
}
total_calls = PLAN_CALLS + sum(v["calls"] for v in per_track_hardcoded.values()) + COMPOSE_CALLS
total_hops = sum(v["hops"] for v in per_track_hardcoded.values())
print("total_model_calls:", total_calls)
print("total_hops:", total_hops)

total_model_calls: 15
total_hops: 10

Exercise 2: How many rounds does a PLAN with three uneven tracks save? (Medium)

Using the same three tracks from Exercise 1 (internal calls 8, 2, and 3), calculate the sequential and parallel rounds, and the percentage saved.

call_counts = [8, 2, 3]
surrounding = PLAN_CALLS + COMPOSE_CALLS
sequential_rounds = sum(call_counts) + surrounding
parallel_rounds = max(call_counts) + surrounding
saved = sequential_rounds - parallel_rounds
pct = round(100 * saved / sequential_rounds)
print(f"sequential={sequential_rounds} parallel={parallel_rounds} saved={saved} ({pct}%)")

sequential=15 parallel=10 saved=5 (33%)

Exercise 3: Why doesn't coordination_cost need to know HOW MANY distinct specialists there are? (Hard)

Without running code, explain why coordination_cost calculates the same DISPATCH_HOPS = 2 for a track regardless of whether its internal specialist is booking_agent, policy_agent, or pricing_agent — and why that's a design advantage, not a limitation.

Summary and next step

• coordination_cost generalizes M3, M4, and M5's three cost formulas into a single function that works over any PLAN of tracks, regardless of how many or which pattern.
• Tested against Demo A: 8 calls, 4 hops, 0 rounds saved (a single track has nobody to share coordination time with).
• Tested against Demo B: 6 calls, 4 hops, 2 rounds saved (33% less, the same pattern M4 L07 already measured).
• The complete system pays a real cost —PLAN_CALLS and DISPATCH_HOPS— that M3/M4/M5's isolated lessons didn't pay, because in the complete system the right pattern gets decided, not assumed in advance.

Next lesson: 08 — Project: Reservo's Multi-Agent System. The capstone: this module's six pieces, assembled into a single system, with the third demo (Valentina, all three patterns combined) and the close of the whole guide.

Additional resources

1. Anthropic — Multi-agent research system — Anthropic documents right there that their multi-agent systems consume considerably more tokens than a single-agent conversation — the same class of coordination cost this lesson quantifies, at token scale instead of calls.
2. Python — Built-in functions sum() and max() — The two functions behind sequential_rounds and parallel_rounds, unchanged since M4 L07.
3. Python — Dataclasses — The structure behind Track, the type coordination_cost receives as its PLAN.
4. Anthropic — Building effective agents — Anthropic's explicit warning about the latency and call cost of multi-agent systems — the same principle this lesson turns into a reusable function.