5. HALF_OPEN and automatic recovery

Overview

Lesson 4 showed how the breaker enters protection: it counts five failures in a row and jumps to OPEN. But a breaker that only knew how to open would be half a tool —worse, it would be a trap—. If it stayed OPEN forever, it would cut off shipping even after it revived, and you'd need a human on call to reset it by hand every time. This lesson builds and measures the other half, the one that makes the breaker an autonomous tool: the HALF_OPEN state and automatic recovery. You're going to see, in the real trace, how the breaker asks itself "has shipping revived yet?" every so often, how it bounces between OPEN and HALF_OPEN while the answer is "no," and how it closes on its own —without anyone touching anything— four seconds after shipping comes back.

The mechanism is a careful probe. When the breaker has been OPEN for a while —the cooldown—, it doesn't reopen the circuit wide (that would be foolish if shipping is still dead: all the accumulated traffic would land on it at once). Instead, it moves to HALF_OPEN and lets a single call through, like someone dipping a finger in the water to see if it's still cold. That single call decides the future: if it succeeds, shipping revived, and the breaker goes back to CLOSED (normal circuit, all the traffic again). If it fails, shipping is still dead, and the breaker goes back to OPEN to wait another cooldown before probing again. A probe, not a reopening; and the decision to close or reopen depends entirely on that probe.

Connection with the module: this lesson closes the cycle lesson 4 opened. Lesson 4 was the "outbound" half (CLOSED → OPEN); this is the "return" one (OPEN → HALF_OPEN → CLOSED, with the HALF_OPEN → OPEN bounce while it's still dead). Together, you have the complete and autonomous cycle. The cooldown we use here fixed at 10 seconds is the second knob lesson 7 teaches you to turn (the first was the failure_threshold). The boundary: here the breaker probes and recovers on its own; what orders does while the breaker is OPEN waiting (fail the order, or degrade by deferring the shipment) is module 7.

The analogy: the smoke detector that rearms itself

Imagine a smoke detector a bit smarter than the one in your house. It detects smoke and sounds the alarm —that's opening the circuit, cutting—. But instead of sounding forever until someone turns it off by hand, it does something clever: every five minutes, it goes quiet for an instant and smells the air just once. If it still smells smoke, it sounds again and waits another five minutes. If the air is already clean, it turns off on its own and returns to its normal watch.

That "smelling every five minutes" is exactly HALF_OPEN. The detector doesn't stay sounding forever (it doesn't stay OPEN forever), but it doesn't turn off blindly trusting that the fire has passed either (it doesn't go back to CLOSED without checking). It tests: it smells once. And there are two details of the detector that are key and that the breaker copies. First, it smells just once, it doesn't throw open all the windows at once: if the fire is still there, opening everything would make things worse —the probe is minimal on purpose—. Second, if it still smells smoke, it restarts its five-minute clock: it doesn't smell frantically every second; it waits a prudent interval between tests, because a fire doesn't go out in a second and smelling every second would only drain the battery.

The breaker is that detector. "Smelling the air once" is letting a probe through in HALF_OPEN. "Clean air → turning off" is HALF_OPEN → CLOSED (revived). "Still smells smoke → sound again and restart the clock" is HALF_OPEN → OPEN (still dead, and opened_at is updated to start the cooldown over). A smoke detector that rearms itself when the danger has passed —that's the circuit breaker's automatic recovery—.

The recovery code

The recovery lives in two places of the CircuitBreaker you already built. First, the entry to HALF_OPEN, in call: when we're OPEN and the cooldown has already passed, we let the probe through.

    def call(self, fn):
        now = self.clock()
        if self.state == OPEN:
            if now - self.opened_at >= self.cooldown_s:
                self._to(HALF_OPEN)      # cooldown passed: ONE probe passes
            else:
                self.rejected += 1        # still in cooldown: reject
                raise CircuitOpenError("circuit is OPEN")
        # in HALF_OPEN (or CLOSED) the call passes to shipping:
        self.allowed += 1
        try:
            result = fn()
        except Exception:
            self._on_failure()
            raise
        else:
            self._on_success()
            return result

The condition now - self.opened_at >= self.cooldown_s is the detector's "every five minutes": it measures how long it's been OPEN since opened_at, and when it reaches the cooldown, it moves to HALF_OPEN and lets that call through (the probe). Notice that only one call enters this way: as soon as it moves to HALF_OPEN, the next decision (success or failure) takes it out of HALF_OPEN, back to CLOSED or to OPEN. There's no way for two calls to pass simultaneously as probes in this sequential version —and in a concurrent version, a lock is added to guarantee exactly one probe—.

Second, the decision based on the probe's result, in _on_success and _on_failure:

    def _on_success(self):
        if self.state == HALF_OPEN:
            self._to(CLOSED)      # the probe worked: shipping revived -> normal
        self.failure_count = 0

    def _on_failure(self):
        if self.state == HALF_OPEN:
            self._to(OPEN)        # the probe failed: still dead -> reopen
        else:
            self.failure_count += 1
            if self.failure_count >= self.failure_threshold:
                self._to(OPEN)

Here's the detector's fork. If the HALF_OPEN probe succeeds, _on_success closes the circuit (HALF_OPEN → CLOSED): clean air, it turns off. If the probe fails, _on_failure reopens (HALF_OPEN → OPEN): still smells smoke, it sounds again. And on reopening, remember that _to(OPEN) updates opened_at = self.clock() —it restarts the cooldown clock—, which is the detector's "restart its five-minute clock." So, after a failed probe, the breaker waits another full cooldown before probing again. Without that restart, the breaker would probe on every call after the first cooldown, and would lose half its point.

The recovery trace, measured

Now the part I want you to see whole: how the breaker bounces while shipping is still dead and how it closes on its own when it revives. Let's recall the scenario: shipping down from second 10 to 60, revives at 60; breaker with failure_threshold=5, cooldown=10s.

What to expect. The complete transitions trace (real output, fixed seed):

--- transitions (t, from -> to) ---
    t= 14.0s      CLOSED -> OPEN        (5 failures: jumps, opened_at=14)
    t= 24.0s        OPEN -> HALF_OPEN   (14+10=24: cooldown passes, probe)
    t= 24.0s   HALF_OPEN -> OPEN        (probe FAILS: shipping still dead, opened_at=24)
    t= 34.0s        OPEN -> HALF_OPEN   (24+10=34: another probe)
    t= 34.0s   HALF_OPEN -> OPEN        (fails, opened_at=34)
    t= 44.0s        OPEN -> HALF_OPEN   (another probe)
    t= 44.0s   HALF_OPEN -> OPEN        (fails, opened_at=44)
    t= 54.0s        OPEN -> HALF_OPEN   (another probe)
    t= 54.0s   HALF_OPEN -> OPEN        (fails, opened_at=54)
    t= 64.0s        OPEN -> HALF_OPEN   (54+10=64: probe... shipping revived at t=60)
    t= 64.0s   HALF_OPEN -> CLOSED      (probe SUCCESS: recovered, no intervention)

Read it as the story it tells. At t=14 the breaker opens (lesson 4). During the next 10 seconds it rejects everything. At t=24 —exactly 10 seconds after opened_at=14— the cooldown expires: it moves to HALF_OPEN and lets a probe through. But shipping is still dead (we're inside the outage [10, 60)), so the probe fails and the breaker goes back to OPEN, with opened_at updated to 24. The clock starts over. Ten seconds later, at t=34, another probe —fails, reopens—. And so on at t=44 and t=54: every 10 seconds, a lone probe touches shipping, confirms it's still dead, and the breaker goes back to waiting. Between probe and probe, all the other calls are rejected in ~0 ms. The breaker is "sounding" (protecting) but "smelling the air" every 10 seconds just in case.

The climax is at t=64. shipping revived at t=60, but the breaker doesn't know it yet —it's been OPEN since opened_at=54, waiting for its cooldown—. At t=64 (54 + 10) the cooldown expires, it moves to HALF_OPEN, and lets a probe through. This time the probe succeeds: shipping responds in 20 ms. The breaker interprets "revived" and closes: HALF_OPEN → CLOSED. From then on, all the traffic passes normally again. Nobody touched anything. No human saw the alert, no engineer reset the breaker. shipping went down, the breaker protected for 50 seconds, probed every 10, and closed on its own 4 seconds after the service came back.

That 4-second delay between the real revival (t=60) and the close (t=64) isn't a defect: it's the price of the cooldown. The breaker can't know shipping revived without probing, and it only probes every cooldown seconds. In the worst case, the service revives just after a failed probe and the breaker takes almost a full cooldown to find out. That's the cooldown trade-off lesson 7 measures: a short cooldown finds out sooner (less recovery delay) but wastes more probes against a still-dead service; a long one wastes fewer probes but takes longer to notice it came back.

Why a single probe, and not fully reopening

It's worth insisting on why HALF_OPEN lets one call through and doesn't reopen the circuit wide, because it's the design decision that avoids a subtle disaster.

Imagine that, when the cooldown expired, the breaker went straight back to CLOSED —letting all the traffic through again—. If shipping revived, perfect. But if shipping is still dead (or worse, if it revived but is fragile, barely getting up), all the traffic accumulated during the cooldown would land on it at once. A service that's just starting to recover, suddenly hit by 10 seconds of pent-up demand, goes down again instantly —and the breaker, which just reopened, has to count another 5 failures to close again—. It would be the module 3 "retry storm" in disguise: the synchronized reopening kills the one that was recovering.

HALF_OPEN avoids exactly that by sending a single probe. If shipping is still dead, only one call checks it and fails —the rest stays protected—. If shipping revived but fragile, that single probe is a minimal load it can handle, and only after confirming it responded well does the breaker open the way to everyone. The single probe is a low-risk test: the "is it OK yet?" information at the lowest possible cost, without risking reopening the flood. It's the difference between dipping a finger in the water and diving in headfirst.

A nuance that production libraries add and that's worth knowing: some let through not one but a small number of probes in HALF_OPEN (say, allow 3 test calls and close only if all 3 succeed), so as not to close over a single lucky success. It's the same idea with more evidence: instead of "one probe decides," "a few probes decide." Our single-probe version is the minimal correct one; the generalization to N probes is a refinement, not a change of concept.

Common mistakes

Not restarting the cooldown clock after a failed probe. What happens: on going back HALF_OPEN → OPEN, opened_at isn't updated, so the breaker thinks the "cooldown is still expired" and lets a probe through on every following call. Why it happens: it's forgotten that reopening should restart the timer. How to spot it: after the first cooldown, the breaker probes shipping on every call instead of every cooldown seconds —it goes back to hitting the dead one repeatedly, losing half the benefit—. How to fix it: _to(OPEN) should update opened_at = clock() whenever OPEN is entered, including the bounce from HALF_OPEN. So, after a failed probe, another full cooldown is waited. Our version does it in _to.

Closing the circuit without probing (going straight back to CLOSED after the cooldown). What happens: on the cooldown expiring, it jumps straight to CLOSED without passing through HALF_OPEN. Why it happens: it seems simpler to "assume it already recovered and go back to normal." How to spot it: if shipping is still dead when the cooldown expires, all the accumulated traffic lands on it at once and knocks it back down —a synchronized reopening, retry-storm style—. How to fix it: never close blindly; pass through HALF_OPEN and let one probe confirm the recovery before opening the way to everyone. The HALF_OPEN state exists precisely so as not to reopen without evidence.

Interpreting the recovery delay as a breaker failure. What happens: it's noticed that shipping revived at t=60 but the breaker didn't close until t=64, and it's reported as "the breaker took too long to realize." Why it happens: instant detection of the recovery is expected. How to spot it: your criticism is "there were 4 seconds in which shipping was healthy but the breaker kept rejecting." How to fix it: that delay is the cooldown doing its job —the breaker only probes every cooldown seconds, so in the worst case it takes almost a cooldown to notice the revival—. It's a conscious trade-off, not a bug: probing more often would reduce the delay but would waste more probes against a still-dead service. If that delay matters a lot to you, you lower the cooldown (lesson 7); there's no way to detect the recovery without some delay, because you have to probe.

Exercises

Exercise 1 — Count the probes. In the trace, the breaker probes at t=24, 34, 44, 54 (all fail) and t=64 (succeeds). (a) Why exactly at those instants and not others? (b) How many "wasted" probes (against a still-dead shipping) were there, and what does that number depend on?

See solution

(a) Each probe occurs exactly one cooldown (10 s) after the last opened_at. The first OPEN was at t=14, so the first probe is at 14+10=24. Since that one fails and reopens with opened_at=24, the next is at 24+10=34. And so on: 34→44, 44→54, 54→64. The instants are 14+10, and then every 10 from there, because each failed probe restarts the clock. The t=64 probe succeeds because shipping revived at t=60, before that probe.

(b) There were 4 wasted probes (t=24, 34, 44, 54), each touching a still-dead shipping. The number depends on two things: the duration of the outage and the cooldown. Approximately: wasted_probes ≈ (outage_duration_after_opening) / cooldown. The outage lasts until t=60, the breaker opened at t=14, so there are ~46 s of outage under protection, and at one probe every 10 s that comes out to ~4-5 probes before it revives. With a shorter cooldown there would be more wasted probes (it would probe more often against the dead one); with a longer one, fewer. That's exactly the cooldown trade-off of lesson 7: fewer wasted probes costs more delay in detecting the recovery.

Exercise 2 — The worst moment to revive. shipping revives at some instant during the outage. (a) What's the best moment for it to revive, in terms of how quickly the breaker detects it? (b) And the worst? (c) With cooldown=10s, what's the maximum possible delay between shipping reviving and the breaker closing?

See solution

(a) The best moment: for shipping to revive just before a scheduled probe. If it revived at, say, t=53.9s (just before the t=54 probe), the t=54 probe would detect it almost immediately —delay of ~0.1 s—. The breaker would close at t=54 instead of t=64.

(b) The worst moment: for shipping to revive just after a failed probe. If it revived at t=54.1s (a hair after the t=54 probe failed and reopened with opened_at=54), the breaker wouldn't probe again until t=64 —it would have to wait almost the full cooldown—, even though shipping was healthy from t=54.1. Those ~10 seconds of "healthy but rejected" are the worst case.

(c) The maximum delay is almost a full cooldown: ~10 seconds (minus an instant). The breaker probes every 10 s, so if the revival falls just after a probe, it takes until the next one —up to 10 s—. The minimum delay is ~0 (if it revives just before a probe). On average, the delay is half a cooldown (~5 s). The practical lesson: the cooldown bounds the recovery delay in its worst case. If your SLA requires recovering within <3 s of the dependency coming back, your cooldown can't be greater than ~3 s —at the cost of more wasted probes—. That calculation is lesson 7.

Exercise 3 — The fragile service. shipping revives but is fragile: it can serve one call per second, but if 10 land on it at once it goes down again. Explain why the single probe of HALF_OPEN handles this case well, and what would happen if, instead of one probe, the breaker reopened straight to CLOSED on the cooldown expiring.

See solution

With the single probe (HALF_OPEN): on the cooldown expiring, the breaker lets a single call through. Fragile shipping can serve that one (it holds 1/s), so the probe succeeds, and the breaker closes. Now —an important nuance— after closing, all the traffic returns; if that traffic is greater than what fragile shipping holds, it would go down again and the breaker would reopen after 5 failures. But at least the probe didn't knock it down: it tested with minimal load. On a service that recovers gradually, the single probe gives it the chance to prove it can handle something before receiving everything.

If the breaker reopened straight to CLOSED: on the cooldown expiring, all the traffic accumulated during the 10 seconds of OPEN would land on shipping at once —many more than 10 simultaneous calls—. Fragile shipping, which only holds 1/s, goes down again instantly under that avalanche. The breaker would have to count 5 failures again and reopen, and the cycle would repeat: reopening → avalanche → relapse → reopening. It's the module 3 retry storm in breaker form: the synchronized reopening kills the one that was recovering.

The lesson: HALF_OPEN with a single probe is what lets a service recover gradually without the reopening knocking it back down. It's the difference between letting one test person into the room and throwing the doors open to the crowd that was waiting outside. (For the case of the service that holds little traffic even when healthy, the complementary tool is limiting how much traffic is sent to it —rate limiting or the module 6 bulkhead—; the breaker handles the total outage, not the reduced capacity.)

Summary and next step

In this lesson you closed the breaker's cycle. You saw how HALF_OPEN does automatic recovery: after the cooldown, the breaker lets one probe through; if it fails it goes back to OPEN and restarts the clock, if it succeeds it goes back to CLOSED. You measured it in the real trace: the breaker bounces OPEN ↔ HALF_OPEN every 10 seconds while shipping is still dead (t=24, 34, 44, 54) and closes on its own at t=64, four seconds after shipping revived at t=60, without anyone intervening. You understood why the probe is a single one (so as not to reopen the flood on a service that's barely getting up) and why the recovery delay (up to a cooldown in the worst case) is a conscious trade-off, not a bug.

Before moving on you should be able to: walk through the recovery trace and explain each bounce; justify why the probe is single and why reopening straight to CLOSED would be dangerous; and compute the recovery delay as a function of the cooldown.

With the complete cycle mastered —opening by threshold (lesson 4), recovering by probe (this one)—, two things remain to close the module. First, a conceptual distinction that orders everything: why the breaker is different from the retry, even though both react to failures. The retry insists; the breaker gives up. Lesson 6 sets them face to face and shows why they combine instead of competing. After that, lesson 7 teaches you to choose the two numbers you've used fixed —failure_threshold=5 and cooldown=10s— with judgment and data, not by eye.

Resources

• Martin Fowler, "CircuitBreaker" — martinfowler.com/bliki/CircuitBreaker.html. Describes the HALF_OPEN state as the recovery mechanism and the idea of the "trial call"; the direct complement of this lesson. Free and in English.
• Michael T. Nygard, Release It!, 2nd ed. (Pragmatic Bookshelf, 2018) — the Circuit Breaker chapter introduces the controlled retry after a wait time and why the reopening should be cautious; the basis of the HALF_OPEN design. In English.
• resilience4j documentation: "CircuitBreaker" — resilience4j.readme.io/docs/circuitbreaker. Explains the real HALF_OPEN, including permittedNumberOfCallsInHalfOpenState (letting N probes through instead of one) and waitDurationInOpenState (our cooldown); the production generalization of what we build here. In English.
• Polly (.NET) documentation: Circuit Breaker pattern — learn.microsoft.com/dotnet/architecture/microservices/implement-resilient-applications/implement-circuit-breaker-pattern. Shows the half-open state and the break duration (durationOfBreak, our cooldown) with the same probe logic. In English.