6. The hidden cost: the flow you can no longer trace with your finger

Description

By the end of this lesson you'll have something almost no material about events gives you: the price, measured, receipt in hand. You'll live through a real Boletia incident at three in the morning, step by step, with the decoupled system we built in lessons 3 and 5, and you'll see exactly the moment a programmer's most basic skill breaks: tracing the flow with your finger.

You'll walk away with three things. First, a precise inventory of the four skills you lose when you go from direct calls to events — and they're four skills you use every day without realizing you have them. Second, the four mitigations that actually work, with their code, because the cost can be reduced a lot if it's done on purpose and from the start. And third, the list of false mitigations: things everyone tries, that feel responsible, and don't work, because they age badly or depend on someone remembering.

This is, to me, the module's most important lesson, and I want to say why. Patterns almost always get taught from the moment they're written, when the author has all the context in their head and the design looks clean. They're almost never taught from the moment they're read, which is where software spends most of its life: someone who didn't write it, months later, under pressure and with no context. A pattern that's written well and read badly is debt with compound interest, and this family generates the most of that kind of debt.

Connection to the module: lessons 2 through 5 built the decoupled system and measured its gains. This lesson hands over the bill. It's lesson 3's necessary complement: there we counted that the flow went from one file to seven, here you're going to feel what that seven means. And it directly sets up lesson 7, which uses this cost as the main argument for not turning into an event what doesn't need it. Lesson 8's project is going to require you to deliver a registry of who listens to what, precisely so you don't lose what you're about to watch get lost here.

The house where nobody knows which switch turns on what

Think of an old house that's been through three remodels. In the hallway there's a panel with four switches. Nobody knows what each one does.

You flip them all and see what turns on. One turns on the hallway light, sure. Another does nothing visible — it turns out it powers the rooftop water pump, which you discover three days later when there's no water pressure. The third turns on the utility room light, which you can't see from the hallway. And the fourth is wired to an outlet in the garage that an electrician put in back in 2011, with no record left behind.

Notice what happened there. The wiring works perfectly. Every wire is correctly placed, everything turns on what it's supposed to turn on, there's no short circuit. The problem isn't functional: it's that there's no blueprint. And without a blueprint, every change gets made blind: you want to put in a new light and don't know which circuit to hang it off; a room's light goes out and you don't know what to check; you sell the house and the buyer inherits the mystery.

Now think of the house next door, built yesterday, with its breaker panel labeled: "circuit 3 — bedrooms and hallway." The wires are just as invisible inside the wall. The difference isn't in the wiring: it's that someone wrote down what feeds what, and kept it updated.

A system with events is a house with wires inside the walls. The decoupling is real and it's good: you can add a circuit without breaking the others. But if nobody labels the panel, every change and every failure turn into an expedition. And here's the part that has to be said in full: the label isn't optional, it's part of the pattern's cost. If your budget for introducing events doesn't include the work of keeping the panel labeled, your budget is incomplete and the system is going to degrade with nobody making a single bad decision.

Worked example: 3:14 a.m.

The phone rings. It's Boletia's founder. The organizer of the year's biggest festival is furious: they've gone two days without receiving the sale notices and found out because an attendee asked why their dashboard showed fewer tickets than they'd sold.

You log into the system. Let's do the diagnosis twice: first with lesson 1's system, the ugly twenty lines, and then with lesson 3's decoupled system. Timing it.

Diagnosis with direct calls.

You open checkout/checkout.py. Scroll down to section 4. There it is:

    organizer = repository.get_customer(event.organizer_id)
    email_channel.send(organizer.email, build_organizer_alert(order, event))

Two lines, and every hypothesis in view: either event.organizer_id points to another customer, or the organizer's email is empty or wrong, or the send failed. You query the database, see the organizer has a valid email, and search the error log for smtp. You find it: the organizer's email has been on a bounce list for two days, because their mailbox filled up.

One file. Three hypotheses. Eight minutes. The ugly system let itself be diagnosed in eight minutes.

Diagnosis with events.

You open checkout/checkout.py. Scroll down to section 4. There it is:

    bus.publish(OrderCompleted(
        order_id=order.id,
        customer_id=order.customer_id,
        event_id=event.id,
        ticket_ids=tuple(t.id for t in tickets),
        subtotal_mxn=order.subtotal,
        service_fee_mxn=order.fee,
        total_charged_mxn=order.total,
        occurred_at=now(),
    ))

And now tell me: does the organizer get a notice? This file doesn't know. There's nothing here that says so. Maybe it does, maybe the feature never existed, maybe it existed and someone removed it last month. You're looking at the code that produces the fact and have no way of knowing what happens next. That's the exact moment the finger on the screen breaks, and it's this lesson's topic.

You go to bus/wiring.py — if you know it exists — and find the subscriptions. Good, send_organizer_alert is registered. Now you have to verify three things that in the previous system weren't questions:

1. Did wire_everything() get called? If someone touched app.py and the call ended up inside a configuration if, the system starts up with no subscriptions and doesn't complain about anything: publishing into the void is a normal case.
2. Did the handler run and fail? The bus isolates and logs. You search the log. You find four thousand lines of "send_organizer_alert failed handling OrderCompleted" and none of them say for which order, because the logging we wrote in lesson 2 only puts the handler's name and the event's.
3. Is it the handler or the channel? The handler calls EmailChannel().send(...), which in turn sits inside notify() for the buyer's case but not for the organizer's. You have to open three files to rebuild the chain.

And if you also moved to queued commands, like in lesson 5, a fourth question gets added: did the job get queued and never run? Now you have to look at the pending-jobs table, see if there are stuck rows, and find out whether the process that consumes them is alive. That process, by the way, isn't on the same machine.

Four files, one database table, a separate process, and six hypotheses. Forty minutes, at three in the morning. And that's with luck.

What to expect from this comparison. The first thing: notice that the root cause was the same in both cases — the organizer's mailbox was full. The decoupled system didn't cause the problem. What it did was multiply the time to find it by five, and add three hypotheses that have nothing to do with the business and everything to do with the machinery. That's the cost, in its purest form: not new failures, but longer diagnoses of the same failures.

The second thing, and I want you to look at it closely because it's what makes this cost treacherous: it gets paid at the worst possible moment. It doesn't get paid while you're writing, when you have time and context. It gets paid during an incident, at night, with someone waiting, and with half your reasoning capacity. A cost that only gets paid at the worst moments is systematically underestimated, because nobody's taking notes when they pay it.

And the third, the good news: almost all of that cost is avoidable, and it's avoidable with work you do once. The four mitigations coming up would have brought those forty minutes down to ten. None of them is complicated. All of them need to be decided at the start, because adding them later, once there are already fifteen subscribers, is much more expensive.

The four skills you lose

It's worth naming them separately, because you use them every day without knowing you have them and only miss them once they're gone.

1. Reading the whole flow in one place. Before, the question "what happens when a purchase completes?" got answered by reading a function top to bottom. Now it gets answered by gathering information from several files, and only if you know which ones. This is the big loss and almost everything else follows from it.

2. Finding who uses something. Your editor has a function called "find usages" or "find references" and you use it without thinking. That function follows the program's call graph. The bus breaks that graph: if you search for uses of send_organizer_alert, the editor's going to show you one line in wiring.py and nothing else, because nobody calls it by name. And the other way around: if you're in checkout.py and want to know who reacts, there's no reference to follow. The tool isn't broken; it's that the information you're looking for is no longer in the code, it's in the data structure the code builds at startup.

3. Reading the call stack. With synchronous dispatch the stack survives, but full of noise: checkout → publish → handler, with the bus in the middle of everything. With asynchronous dispatch or with queued commands, the stack gets cut: the handler's error happens in another process, at another moment, and its stack doesn't mention the purchase that triggered it. The question "where did this come from?" stops having a technical answer and starts depending on whether someone put in a correlation identifier.

4. Trusting that what you read is everything that happens. This is the subtlest and most dangerous one. Before, if you read the whole function, you knew you'd read everything. Now, reading doesn't give you certainty: there can always be a subscriber you didn't see, registered from a module you didn't know existed. You go from "I know what this code does" to "I know what this code does, plus whatever whoever's listening does." And that uncertainty colors every change you make afterward.

The four mitigations that work

Mitigation 1 — A central registry that can be read and can't go stale.

Lesson 3's wiring.py is already half the solution. The other half is making the system able to tell you its own wiring, so the information doesn't depend on someone writing it down:

# File: bus/bus.py — added to EventBus

    def describe(self) -> str:
        """Prints the full wiring. Run by hand or at startup.

        This function is the house's labeled panel. Its value isn't in
        being sophisticated — it's ten lines — but in the information
        coming from the real bus and not from a document someone
        maintains by hand. A document goes stale; this can't.
        """
        lines = []
        for event_type, handlers in sorted(
            self._subscribers.items(), key=lambda kv: kv[0].__name__
        ):
            lines.append(f"{event_type.__name__}:")
            for handler in handlers:
                module = handler.__module__
                doc = (handler.__doc__ or "").strip().split("\n")[0]
                lines.append(f"  - {module}.{handler.__name__}  # {doc}")
        return "\n".join(lines)

With that, python -m bus.describe gives you:

OrderCompleted:
  - inventory.subscribers.mark_tickets_sold        # Marks tickets as sold...
  - notifications.subscribers.send_buyer_confirmation  # Confirmation to the buyer...
  - notifications.subscribers.send_organizer_alert     # Notice to the organizer. Email only.
  - billing.subscribers.issue_invoice              # Issues the invoice and sends it...
  - analytics.subscribers.track_order_paid         # Logs the sale and refreshes...
  - loyalty.subscribers.add_loyalty_points         # One point for every ten pesos.

The incident's forty minutes start coming down right here: the question "does the organizer get a notice?" gets answered in fifteen seconds, and with the certainty that the answer comes from the live system.

And the finishing touch, which is what keeps this from degrading:

# File: tests/test_wiring.py

def test_the_wiring_matches_the_documented_map():
    """The versioned map has to match the real wiring.

    If someone adds a subscriber and doesn't update docs/event-map.txt,
    this test fails and tells them exactly what to run to fix it. That
    way the document can't go stale: the system doesn't allow it.
    """
    wire_everything()
    documented = Path("docs/event-map.txt").read_text().strip()
    assert bus.describe() == documented, (
        "The wiring changed. Run `python -m bus.describe > docs/event-map.txt` "
        "and commit it with your change."
    )

It's a twelve-line test and it does something no amount of good will achieves: it turns keeping the map up to date into a mechanical requirement instead of a discipline. The entire difference between documentation that's useful and documentation that lies lies in whether the system verifies it.

Mitigation 2 — Traces with an identifier that runs through everything.

The problem of the log with four thousand useless lines gets fixed with an old, cheap idea: every operation carries a unique identifier that travels with it everywhere.

# File: bus/events.py

@dataclass(frozen=True)
class OrderCompleted:
    order_id: int
    # ... the rest of the fields ...
    trace_id: str            # the thread connecting everything that caused this purchase

# File: bus/bus.py

    def publish(self, event):
        name = type(event).__name__
        handlers = self._subscribers[type(event)]
        trace = getattr(event, "trace_id", "-")
        self._logger.info("publish %s trace=%s handlers=%d", name, trace, len(handlers))

        for handler in handlers:
            started = time.monotonic()
            try:
                handler(event)
                self._logger.info(
                    "  ok %s trace=%s ms=%d",
                    handler.__name__, trace, (time.monotonic() - started) * 1000,
                )
            except Exception:
                # The handler's name AND the trace identifier.
                # Without the second one, the log says something failed
                # but not for whom, and it's exactly as useless as nothing.
                self._logger.exception(
                    "  FAILED %s trace=%s", handler.__name__, trace,
                )

And the log goes from this:

ERROR  send_organizer_alert failed handling OrderCompleted
ERROR  send_organizer_alert failed handling OrderCompleted
ERROR  send_organizer_alert failed handling OrderCompleted

to this:

INFO   publish OrderCompleted trace=7f3a9b handlers=6
INFO     ok mark_tickets_sold trace=7f3a9b ms=12
INFO     ok send_buyer_confirmation trace=7f3a9b ms=340
ERROR    FAILED send_organizer_alert trace=7f3a9b
         SMTPRecipientsRefused: 552 mailbox full — organizer@festival.mx
INFO     ok issue_invoice trace=7f3a9b ms=88
INFO     ok track_order_paid trace=7f3a9b ms=5
INFO     ok add_loyalty_points trace=7f3a9b ms=3

That block answers, at a glance, the incident's six hypotheses: it got published, there were six registered subscribers, all six ran, one failed, and the error message literally states the root cause. Forty minutes turn into two.

The trace identifier has to be born in the HTTP request — in api/routes.py — and travel through the event, through the queued command, and into every log line. If the command goes to a queue, the identifier travels with it, and that way the job that runs half an hour later in another process stays connected to the purchase that triggered it. It's the only way to repair skill number 3 from the list above.

Mitigation 3 — Names that say what happens, not how it's built.

This one costs nothing and always gets neglected. Three rules:

• No anonymous functions in the wiring. bus.subscribe(OrderCompleted, lambda e: ...) produces a log that says <lambda> and a useless map. Every subscription is a named function.
• The handler's name says what it does, not that it's a handler. send_organizer_alert, not on_order_completed or handle_order. When five handlers for the same event are all called on_order_completed across five different modules, the log and the map stop being useful.
• The handler's first comment line is its description on the map. describe() already uses it. It's a concrete reason to write it well.

Mitigation 4 — A ceiling on the number of events.

The least technical one, and the one that most decides the outcome over two years. A team agreement, written right in bus/events.py:

"""Boletia domain events.

TEAM AGREEMENT (revisited every quarter):
  - Maximum 8 event types in the whole system. Today there are 4.
  - A new event requires at least TWO expected subscribers.
    With just one, it's a direct call in disguise.
  - An event that's left with a single subscriber for a quarter
    gets retired and turned into a direct call.
"""

No tool enforces that comment, and it still works, because it turns "should we add an event?" into an explicit conversation with a shared criterion, instead of an invisible individual decision. A bus with four event types fits in your head. One with forty doesn't, and there's no technical mitigation that fixes that.

The false mitigations

These always get tried, feel responsible, and don't work. Worth knowing so you don't waste effort on them.

A hand-written document. Someone creates docs/event-architecture.md with a beautiful table of who listens to what. It lasts two months. Then someone adds a subscriber in a hurry, doesn't update the document, and from that moment on the document is worse than having nothing: it gives false confidence and sends people in the wrong direction. A document about the code that the code doesn't verify always ends up lying. If you're going to write the map, generate it from the system and verify it with a test, like in mitigation 1.

A comment in checkout listing the subscribers. It's the same trap, closer to the code and therefore more believable. It ages just as badly, with the added downside that whoever reads it is going to trust it more than a separate document.

Relying on the editor's "find usages." It doesn't work through the bus, and that's not a flaw in the editor: the relationship doesn't exist in the code, it exists in the data structure the code builds at startup. No static tool can see it.

Text-searching for the event's name. It helps a little and fails right when you need it most: it finds the subscribe calls written the expected way and misses the ones inside a decorator, the ones that build the name dynamically, and the ones that get registered from a module only imported under certain configuration. A search that works 90% of the time is, during an incident, a source of wrong conclusions.

"Let each person document their own subscriber." It spreads the information across as many places as there are subscribers, which is exactly the problem we're trying to solve.

The pattern behind all five: they all depend on a human remembering, and they all fail silently when someone doesn't. The four that work share the opposite: either they come out of the live system, or there's a test that makes them fail.

The learning cost, which nobody counts

There's one last cost that shows up in no technical measurement and that, on a small team, weighs more than all the others.

Boletia has six people. Before the refactor, someone joining the team could read checkout.py and understand the purchase in an afternoon: it was ugly, it was long, but it was linear and complete. After the refactor they have to understand what a bus is, what an event is, where the wiring lives, why inventory stayed inside and notices didn't, what the command queue is, and why there's a separate process. That's not an afternoon: it's two or three days, and it's a topic they're going to be unsure about for months.

That cost gets paid once per person who touches the system, including all future ones. On a sixty-person team where checkout belongs to one team and notifications to another, it amortizes quickly: the gain from two teams not stepping on each other is huge. On a six-person team where everyone touches everything, it might never amortize.

That calculation — how many people, how separate, how many new interested parties per year — is what really decides whether this family of patterns is worth it for you. It isn't a technical calculation. It's all of lesson 7, and that's why it comes right after this one.

Common mistakes

Introducing the bus without introducing the panel (judgment). What happens: the refactor gets done, it looks nice, it gets deployed, and the mitigations get left "for later." Later never arrives, because there's no pain yet. The pain arrives eight months in, with fifteen subscribers, when adding describe() and the traces is already a project instead of an afternoon. Why it happens: because the refactor has a visible result and the mitigations don't; nobody applauds a trace identifier. How to spot it: if your change adds a bus and doesn't add a single new log line or a way to list the wiring, it's incomplete. How to fix it: treat them as part of the same change, not a future improvement. The rule I'd use: the bus and its describe() get written in the same commit. It's twelve more lines; there's no excuse.

Logging that something failed without logging for what (conceptual). What happens: the bus catches the exception and writes "handler X failed." Months later there are thousands of those lines and none of them help, because they don't say for which order, for which customer, or at what point in the flow. Why it happens: because when writing the except, you think about "leave a record," not about "let someone act on this record at three in the morning." How to spot it: look at one of your system's error lines and ask whether it alone lets you start investigating. If you need something else to know which case it is, the line isn't useful. How to fix it: a trace identifier and the main entity's identifier on every line, always. A log with no correlation identifier is noise with a badge of responsibility.

Believing that decoupling code decouples understanding (conceptual). What happens: the system gets declared "modular" because no file imports another. But answering any business question requires opening seven files, so in practice nobody understands one part without understanding the whole — exactly the opposite of what was intended. Why it happens: from confusing two things with the same name. Compile-time coupling — who imports whom — genuinely went down. Conceptual coupling — how much you need to know about the rest to understand one part — may have gone up. How to spot it: give a business question to someone who didn't write the code and time them. It's the only honest measurement I know for this. How to fix it: it doesn't get fixed by removing the pattern; it gets fixed with the labeled panel. The goal isn't for nobody to need to see the whole thing, it's for seeing the whole thing to cost fifteen seconds instead of forty minutes.

Exercises

Exercise 1 — Reconstruct the flow with each design's tools. A question comes in from legal: "at exactly what moment does a purchase's invoice get issued, and what happens if it fails?" Write the steps you'd take to answer it (a) in lesson 1's system, (b) in the decoupled system with no mitigations, and (c) in the decoupled system with all four mitigations. Estimate each one's time.

Exercise 2 — Design the test that prevents the worst bug. This architecture's hardest bug is wire_everything() not getting called, or getting called halfway, with the system starting up with no subscriptions and not complaining about anything. Write a test that makes it impossible, and explain why the bus shouldn't complain just because it publishes with no subscribers.

# File: tests/test_wiring.py

REQUIRED = {
    OrderCompleted: {
        "mark_tickets_sold", "send_buyer_confirmation", "send_organizer_alert",
        "issue_invoice", "track_order_paid", "add_loyalty_points",
    },
    OrderCancelled: {"release_tickets", "notify_cancellation"},
}


def test_every_required_subscription_is_wired():
    """Fails if someone deletes a subscription or if wire_everything()
    doesn't register everything. It's the net that catches this design's
    quietest bug: a system that starts up listening to nobody and doesn't
    complain."""
    fresh = build_app_bus()          # the same startup path production uses
    for event_type, expected in REQUIRED.items():
        actual = {h.__name__ for h in fresh.subscribers_of(event_type)}
        assert expected == actual, (
            f"{event_type.__name__}: missing {expected - actual}, "
            f"extra {actual - expected}"
        )

Exercise 3 — Put a price on the cost. Your boss asks whether it's worth keeping the event architecture or whether it's better to go back to direct calls. Build an argument with estimated numbers: make up plausible figures for (a) incidents per year involving the purchase flow, (b) extra diagnosis minutes per incident, (c) new interested parties per year, (d) hours saved per new interested party. Conclude.

Summary and next step

In this lesson you handed the decoupled design its bill. You lived through the same incident twice — the organizer who never got their notice — and saw that the root cause was identical while the time to find it multiplied by five, with three new hypotheses that had nothing to do with the business and everything to do with the machinery.

You named the four skills that get lost: reading the whole flow in one place, finding who uses something, reading the call stack, and trusting that what you read is everything that happens. And you learned the four mitigations that genuinely work — a describe() that comes from the live system plus a test that verifies it, traces with a correlation identifier, explicit names, and an agreed-upon ceiling for the number of events — along with the five that don't work, all sharing the same flaw: they depend on a human remembering.

And you saw the cost no technical measurement captures: the learning cost, paid once per person who touches the system, which on a small team might never amortize.

Before moving on you should be able to: explain why the editor's "find usages" stops working with a bus; write the describe() and the test that keeps it honest; tell a real mitigation from a false one with the question "does this depend on someone remembering?"; and argue the pattern's cost in people-hours.

What's next is the natural consequence of all this. If the cost is real and gets paid at the worst moments, the important question stops being "how do I implement Observer?" and becomes "when shouldn't I?" Lesson 7 is the module's vaccine: the cases where a direct call is more honest, easier to debug, and — this matters more than it looks — easier to delete. And the concrete signs that the event genuinely is worth it, so the decision stops being a matter of taste.

Resources

• Martin Fowler — What do you mean by "Event-Driven"? — the final section, on the downsides, matches this lesson point by point and comes from someone who's seen a lot of systems like this.
• Django — Signals — its warning about preferring a direct call when sender and receiver are known is written by people who maintain the framework and see the bug reports.
• Python — logging — for mitigation 2's structured logging. Worth looking at LoggerAdapter and extra, which is the clean way to put the trace identifier into every line without repeating it by hand.
• OpenTelemetry — Traces — the industrial version of the correlation identifier. There's no need to adopt it on a system Boletia's size, but its mental model — one trace, several spans, one parent — is exactly the one you want to have in your head.