8. Project: put a breaker on `shipping` in Mercado and measure

Overview

This is the module's capstone. Until now you learned the circuit breaker piece by piece —why hitting a dead one is expensive, the three states, how it opens by threshold, how it recovers on its own, how it differs from the retry, how to choose its two numbers—. Now you apply it whole, with your own hands, to Mercado's shipping. The task has the shape you already know from the previous modules and that's the backbone of the whole guide: you take an unprotected system, measure its fragility with numbers, apply the pattern, and measure again to prove it worked. Not "I put a breaker because the book says so"; "I put a breaker, and here's the table showing that the calls to a dead shipping went from 50 timeout waits to 45 rejections in ~0 ms, with automatic recovery when it revives."

You're going to work with the call from orders to shipping —the same one you measured raw in lesson 2, the one that burns 100.8 thread-seconds when shipping goes down for 50 seconds—. First you instrument it and confirm its fragility. Then you wrap it in the three-state CircuitBreaker you built in lesson 3, choose failure_threshold and cooldown with the judgment of lesson 7, and measure again with the same seed. The deliverable is the resilient code, the before/after metrics table, and —most importantly— the justification of why each number is what it is and which failure the breaker solves and which it leaves pending for the following modules.

Connection with the module: this lesson doesn't introduce a new concept; it integrates the previous seven into a workflow. Lesson 2 gave you the simulator of shipping going down; lesson 3, the CircuitBreaker; lessons 4 and 5, how it opens and recovers; lesson 6, why retry isn't enough; lesson 7, how to choose the thresholds. Here you put them together. When you finish, you'll have done from start to finish the cycle "measure the fragility → apply the pattern → measure the improvement" on shipping, and you'll be ready for module 6 (bulkhead, isolate the pool) and module 8 (the capstone that combines all the patterns in a single armored call).

The deliverable, at a glance

By the end of the project you'll have produced three things:

1. The instrumented and armored call to shipping (code): the version wrapped in the CircuitBreaker, with failure_threshold and cooldown chosen and justified, and the handling of the CircuitOpenError in orders.
2. The before/after metrics table: calls that touch the dead shipping, calls rejected in ~0 ms (saved), thread-seconds burned, latency of a rejected call vs. a real one, and the instant of automatic recovery —measured WITHOUT and WITH a breaker under the same outage and the same seed—.
3. The justification: why you chose that failure_threshold and that cooldown, which failure the breaker solves (the waste of hitting a dead one and the contagion through busy threads) and which failure it does not solve (shipping is still down; the shipments aren't created), pointing out which later pattern attacks what remains.

Step 1 — Take the fragile call and measure its fragility

We start from lesson 2's simulation: orders calling shipping once per second for 90 seconds, with shipping down from second 10 to 60 (revives at 60) and a timeout of 2 s per call. This is the "fragile" call, without a breaker —each call tries, with no memory of the previous ones—:

# The FRAGILE call: no breaker. Each call tries, even if the previous 40 failed.
def create_shipment_fragile(order):
    return shipping.create(order)      # if shipping is dead, it hangs -> timeout 2 s

Run the simulation without a breaker and note the "before" numbers. They're the ones you already measured in lesson 2, now as your baseline:

=== WITHOUT breaker (the fragile call) ===
  calls that waited for the timeout : 50   (one for each second of the outage)
  thread-seconds burned waiting      : 100.8s
  latency of a call (during outage)  : 2000 ms
  recovery after shipping revives    : immediate but irrelevant
                                       (it kept hitting all through the outage anyway)

Translate this to the left half of your table. During the outage, the 50 calls wait for the full timeout: 100.8 thread-seconds of orders burned waiting for a dead one, with each call taking 2000 ms to fail. orders survives, but pays that cost in busy threads —the seed of lesson 2's contagion—. Save these numbers; they're your "before."

Step 2 — Choose the failure_threshold and the cooldown with judgment

Before wrapping the call, decide the two numbers with the lesson 7 method, not by eye. You need two pieces of data about shipping: how much noise it has (for the threshold) and how long its outages last plus the recovery SLA (for the cooldown).

For this project we assume the realistic data of lesson 7's exercise 3: shipping has little noise (fails in isolation ~1 in 1000), high traffic, outages of 30 s to several minutes, and an SLA of "recover in <20 s." From that:

• failure_threshold = 5. Low, because shipping has little noise: it's unlikely to gather 5 consecutive failures by chance, so the risk of a false positive is minimal, and a low threshold gives fast reaction (the detection toll is only 5 timeouts). With real high traffic, those 5 failures accumulate in fractions of a second.
• cooldown = 10s. Below the ceiling the SLA imposes (~20 s in the worst case ≈ a cooldown), and long enough not to waste too many probes against a shipping that's usually down a while. With cooldown=10 s, in our simulation the breaker wastes 4 probes and recovers 4 s after the revival.

cb = CircuitBreaker(failure_threshold=5, cooldown_s=10.0)

Note the justification now, while you have it fresh: the threshold is low because the dependency is low-noise (the false positive is unlikely) and the detection toll must be kept small; the cooldown is bounded by the recovery SLA and, within that ceiling, raised to waste fewer probes. That justification is part of the deliverable: a threshold without justification is a magic number.

Step 3 — Wrap the call and measure again

Now wrap the call to shipping in the breaker. orders no longer calls shipping bare; it calls it through cb.call, and handles the CircuitOpenError when the breaker is OPEN:

# The ARMORED call: through the breaker. When it's OPEN, it fails in ~0 ms.
def create_shipment_resilient(order):
    try:
        return cb.call(lambda: shipping.create(order))
    except CircuitOpenError:
        # the breaker is OPEN: shipping was NOT touched, it failed in ~0 ms.
        # here orders decides: fail the order, or DEGRADE (defer the shipment -> module 7).
        return None

Run the same simulation (same seed, same outage) with the armored call and note the "after" numbers:

=== WITH breaker (failure_threshold=5, cooldown=10s) ===
  calls that touched shipping (real)  : 9    (5 of the toll + 4 probes)
  calls rejected in ~0 ms (saved)     : 45
  thread-seconds burned waiting       : 18.7s
  latency of a rejected call          : ~0.001 ms
  automatic recovery (HALF_OPEN->CLOSED): t=64s
                                        (shipping revived at t=60; 4 s delay)

This is the right half of your table. The calls that touch the dead shipping dropped from 50 to 9, the ones rejected in ~0 ms rose to 45, the thread-seconds burned dropped from 100.8 to 18.7, and —what the baseline didn't even have— the breaker recovered on its own at second 64, four seconds after shipping revived, without anyone intervening.

Step 4 — Verify the rejection latency with a real clock

The ~0 ms of the rejection latency deserves an honest measurement with a real wall clock, not just the simulated clock. Set up a shipping that actually hangs (a sleep) and measure, with time.perf_counter, how long a real failed call takes vs. one rejected by the OPEN breaker:

import time

def slow_shipping():
    time.sleep(0.05)                  # a real hang (small, so as not to wait 2 s)
    raise TimeoutError("shipping hung")

cb = CircuitBreaker(failure_threshold=3, cooldown_s=100.0)
# ... trip the breaker with 3 failures, then measure the rejection ...

What to expect. Real output (wall clock, time.perf_counter):

=== real latency: call vs rejection ===
  a real call to hung shipping        : ~56 ms   (the sleep + overhead)
  a rejection from OPEN breaker (avg 1000): 0.001 ms
  times shipping was touched after opening : 0

The contrast is five orders of magnitude: a real call to the hung shipping takes ~56 ms (and in production, with the 2 s timeout, it would be 2000 ms); a rejection from the OPEN breaker takes 0.001 ms —a microsecond—, because it doesn't touch shipping, doesn't open a connection, doesn't wait for anything: it just throws CircuitOpenError. And the last line confirms it: after opening, shipping was touched zero times. That "~0 ms" we've cited all module is literal and measured: the rejection is ~50,000 times faster than the real call. That's the physical difference between "failing slowly" and "failing fast."

Step 5 — Build the table and the justification

Put everything together in the before/after table, which is the heart of the deliverable:

And the justification in prose, which accompanies the table:

• What the breaker solves: the waste of hitting a dead one and its contagion. The calls to a downed shipping went from 50 waits of 2 s to 45 rejections of ~0 ms; orders recovered 82.1 thread-seconds that its threads now dedicate to healthy traffic, and shipping stopped receiving traffic (except one probe every 10 s), which gives it a respite to recover. Besides, the breaker recovered on its own when shipping revived —without human intervention—.
• What the breaker does NOT solve: shipping is still down, and the shipments aren't created. The breaker protects orders from exhausting itself and gives shipping a respite, but it doesn't create the shipments that shipping can't create. That's left for module 7 (degradation): when the breaker is OPEN, instead of failing the order, orders can defer the shipment —confirm the purchase and create the shipment later, when shipping comes back—, so the user completes the checkout even though shipping is dead. The breaker decides when to stop calling; the degradation decides what to do with that gap.
• Why each number: the failure_threshold=5 is low because shipping is low-noise (false positive unlikely) and it keeps the detection toll small; the cooldown=10s is below the recovery-SLA ceiling (<20 s) and raised within it to waste fewer probes.

Rubric: how you know you did it right

Your project is complete when you can answer affirmatively to all of this:

• Did you measure the "before" with concrete numbers (calls that wait for the timeout, thread-seconds burned, latency per call), not with a qualitative description?
• Does each threshold (failure_threshold, cooldown) have a justification derived from shipping's data (noise, outage duration, recovery SLA), and isn't a copied number?
• Does your before/after table show the four dimensions —calls saved, thread-seconds, rejection latency, automatic recovery—, not just one?
• Did you measure the automatic recovery (the HALF_OPEN → CLOSED instant after the revival) and not just the opening?
• Did you verify the rejection latency with a real clock (~0 ms) against the real call, so the "~0 ms" isn't a claim but a measurement?
• Did you explicitly distinguish which failure the breaker solves (the waste and the contagion) from which it doesn't (the shipments shipping doesn't create), pointing out that the degradation (module 7) attacks what remains?

If you answer "no" to any of these, there's your next iteration. The measurement is the proof; without it, you didn't do the project, you just wrote code.

Common mistakes in the project

Declaring the breaker a failure because the shipments weren't created. What happens: the student sees that, with a breaker, the 50 orders of the outage still have no shipment and concludes "the breaker didn't work." Why it happens: the wrong metric is measured —the breaker doesn't create shipments, it protects orders—. How to spot it: your conclusion ignores the 82.1 thread-seconds recovered and the 45 fast rejections. How to fix it: the breaker's improvement lives in orders (threads recovered, contagion avoided, automatic recovery) and in shipping (respite), not in the shipments. The absent shipments are a degradation problem (module 7); the breaker solves the waste, not the cause.

Copying failure_threshold=5, cooldown=10s without justifying. What happens: the example's numbers are used without deriving them from shipping's data. Why it happens: they're the ones that appear all through the module. How to spot it: you can't explain why 5 and not 10, nor why 10 and not 30. How to fix it: derive the threshold from the dependency's noise and its traffic, and the cooldown from the duration of its outages and the recovery SLA (lesson 7). The module's defaults are a starting point for measuring, not a final answer —the project is justifying them for your shipping—.

Skipping the recovery measurement. What happens: the saving is measured (calls rejected) but not the automatic recovery (the HALF_OPEN → CLOSED at t=64). Why it happens: the saving is the "star" number and the recovery seems secondary. How to spot it: your table has no "recovery after revival" row. How to fix it: automatic recovery is half the breaker's value —a breaker that opens but doesn't recover on its own would need a human to reset it—. Measure the instant it closes after the revival and the delay relative to it (4 s in our case); that autonomy is what makes the breaker production-ready.

Exercises

Exercise 1 — Reduce the toll. In your table, the breaker touches the dead shipping 9 times (5 of the toll + 4 probes). (a) If you lowered failure_threshold to 3, how would those 9 change? (b) If you raised cooldown to 20 s, how would they change? (c) Which of the two changes has a cost, and what is it?

Exercise 2 — Connect with the degradation. The breaker solves the waste but leaves 50 orders without a shipment. Design, in pseudocode, how orders could degrade when the breaker is OPEN, so the checkout completes anyway. What would you tell the user? What happens with the shipment when shipping revives?

def create_shipment_resilient(order):
    try:
        shipment = cb.call(lambda: shipping.create(order))
        return {"status": "shipped", "shipment": shipment}
    except CircuitOpenError:
        # shipping is dead (breaker OPEN): we do NOT fail the checkout.
        # We enqueue the shipment to create it later, when shipping comes back.
        deferred_queue.push(order)                 # will be processed when shipping revives
        return {"status": "deferred", "message":
                "Your purchase is confirmed. The shipment will be scheduled shortly."}

Exercise 3 — Transfer the pattern. Outside Mercado, describe another system that calls a dependency that can die entirely for a while (for example, a service that queries an external geolocation API, or a backend that calls a push-notifications provider). Apply the project's cycle: what would you measure "before," what failure_threshold and cooldown would you set, and which failure would the breaker solve and which not?

Module close and where it goes next

You finished the circuit breaker module, and you finished it by measuring. You took the fragile call from orders to shipping, quantified its fragility (50 calls waiting for the timeout, 100.8 thread-seconds burned), wrapped it in the three-state CircuitBreaker with justified thresholds, and proved the improvement with a table: 45 calls saved in ~0 ms, 82.1 thread-seconds recovered, shipping with a respite, and automatic recovery at second 64 without anyone intervening. Above all, you practiced the cycle that governs the whole guide —measure → apply → measure— and learned to distinguish what the breaker solves (the waste and the contagion) from what it leaves pending (the shipments shipping doesn't create).

What the breaker leaves pending is the agenda of the following modules, and now it makes sense. The breaker cuts the calls to a dead shipping, but while it was CLOSED waiting for the threshold —and on each HALF_OPEN probe— those calls still hijack threads from orders' shared pool; isolating shipping's pool so that not even those few calls can exhaust payments' threads is the bulkhead (module 6), the breaker's cousin. And what orders does with the order when the breaker rejects —fail hard, or degrade by deferring the shipment so the checkout completes anyway— is graceful degradation (module 7). Finally, combining everything —timeout to bound each attempt, retry for the hiccups, breaker for the outages, bulkhead to isolate, idempotency so retrying doesn't duplicate, degradation to complete the checkout— in Mercado's checkout end to end is the capstone (module 8). The breaker you just mastered is one of the columns of that building.

Summary and next step

In this project you integrated the module's seven concepts into the "measure → apply → measure" cycle on Mercado's shipping. You measured the "before" (50 calls to the timeout, 100.8 thread-seconds, 2000 ms per call), chose the thresholds with judgment (failure_threshold=5 for shipping's little noise, cooldown=10s for the recovery SLA), wrapped the call in the CircuitBreaker, verified the rejection latency with a real clock (~0.001 ms vs ~56 ms), and measured the "after" (9 real calls, 45 rejected, 18.7 thread-seconds, automatic recovery at t=64). The deliverable is the code, the before/after table and the justification of each number and of what the breaker solves and what it doesn't.

The project's deepest lesson is the same as the whole guide's, now with the breaker: a pattern is justified by measuring, not by citing. The before/after table is what turns "I put a breaker" into a defensible engineering decision, and the habit of distinguishing "what the pattern solves" (the waste and the contagion) from "what it leaves pending" (the absent shipments) is what lets you combine patterns with judgment in the capstone instead of piling them up.

With this you master the circuit breaker from start to finish: why hitting a dead one is expensive, the three states, how it opens by threshold and recovers by probe, how it differs from the retry and how they combine, how to choose its thresholds, and how to apply and measure it. What follows is module 6: bulkheads and isolation —how to give shipping its own thread compartment so that its failure, while the breaker reacts, can't exhaust payments' or catalog's threads—. It's the breaker's natural complement: the breaker stops calling the dead one; the bulkhead contains the damage in the meantime.

Resources

• Michael T. Nygard, Release It!, 2nd ed. (Pragmatic Bookshelf, 2018) — the stability patterns chapter shows how the Circuit Breaker combines with timeout, bulkhead and degradation in a real system; the mental framework for this project and the modules that follow. In English.
• Martin Fowler, "CircuitBreaker" — martinfowler.com/bliki/CircuitBreaker.html. The reference for the implementation you just applied; useful to contrast your CircuitBreaker with the reference version. Free and in English.
• resilience4j documentation: "CircuitBreaker" — resilience4j.readme.io/docs/circuitbreaker. The real library with which you'd implement this breaker in production (Java/Kotlin), with metrics and events for the dashboard your before/after table anticipates. In English.
• Polly (.NET) documentation: Circuit Breaker pattern — learn.microsoft.com/dotnet/architecture/microservices/implement-resilient-applications/implement-circuit-breaker-pattern. The .NET equivalent, with the same before/after cycle and guidance to combine it with retry (the bridge to the module 8 capstone). In English.