Module 5: The Incident Lifecycle
6. On-call, with honesty about its cost
Description
Module 1, lesson 7 already defined on-call, quoted from Google SRE: committed availability, with an explicit limit — "no more than 25% [of an engineer's time] can be spent on-call". This lesson doesn't repeat that definition; it extends it with the angle VALIDACION.md flagged as a critical gap across the entire market: the real, financial and operational cost of how that availability gets implemented, with the full Hacker News quote that directly motivates the design of the rotation lesson 7 is going to build.
Connection to the module
This lesson is, on purpose, the most honest one in the module: it doesn't promise that a good on-call rotation solves all of a system's reliability problems. It names, with the same precision this whole guide applies to every representative case, what a well-designed rotation DOES solve and what it DOESN'T — and why the way it's implemented (portable, or locked inside a single tool) matters as much as the shift design itself.
The quote, on the real cost of locking on-call inside a single tool
src/paths/aws-cloud-ecosystem/VALIDACION.md, the same market audit that motivated this whole guide's design, quotes a real Hacker News comment, from user jamiemallers (thread 47088882), about the pricing pattern of commercial on-call platforms and the real risk of depending on just one:
"Switching costs are higher than almost any other category."
— jamiemallers, Hacker News (#47088882)
The full point, verified against the source for this lesson: the comment notes that on-call tools like PagerDuty tend to follow a well-known pricing pattern — low cost at first, to secure adoption, raising the per-seat cost once the tool is already deeply integrated into a team's daily workflow. What makes that pattern work, specifically for on-call tools, is that escalation chains, schedules, and monitoring integrations end up so intertwined with the specific platform that migrating to another tool stops being a simple technical decision and becomes a full process re-engineering project. The comment's concrete recommendation: treat on-call routing as a thin, portable layer — schedules stored in a simple format (the comment mentions YAML), alert routing based on open standards — instead of building the entire workflow directly inside a single vendor's proprietary interface.
THE PATTERN THE QUOTE DESCRIBES
Initial adoption Months/years later The real cost
───────────────── ──────────────────── ──────────────
Low price, Schedules, escalation Migrating requires
easy to integrate chains, and integrations rebuilding that
with existing monitoring now live INSIDE entire process
│ the platform from scratch,
▼ │ on another tool
Team adopts ▼ │
the tool The vendor raises ▼
with confidence the per-seat price "Switching costs
higher than
almost any
other category"What a well-designed on-call rotation DOES solve
A clear, predictable rotation solves a real, concrete problem: the "who do I call?" ambiguity at the exact moment an incident gets declared. Without it, that first minute of a real incident gets spent figuring out who's available, instead of starting to respond. A well-designed rotation also distributes on-call's human cost predictably and auditably — every person knows, weeks in advance, when it's their turn, instead of depending on the goodwill of whoever answers first — and it makes it possible to verify, with data, whether the 25% limit Google SRE already cites is being respected or not.
What an on-call rotation does NOT solve, no matter how well designed
It doesn't reduce how often something breaks. A perfect rotation doesn't improve deployed code quality, nor does it reduce the actual number of ALERTING-POLICY.md alerts that fire — that's engineering work (better tests, better runbooks, this guide's entire Module 7), not shift scheduling. It doesn't eliminate the human cost of being interrupted, it only distributes it more fairly — Google SRE's own 25% limit is, in itself, an admission that on-call has a real cost, even inside the organization that probably manages this process more rigorously than almost any other in the industry. It doesn't substitute for good severity classification — a perfect on-call rotation, combined with a badly designed severity matrix that fires SEV1 for every minor anomaly (the error this module's lesson 3 already named), burns out any team just as fast, no matter how fair the shifts are. And, this lesson's central point: it doesn't protect against the cost of switching tools if the whole on-call process gets built, with no portability layer whatsoever, inside a single vendor's proprietary interface.
Why this matters for lesson 7
This module's lesson 7 builds oncall/schedule.py with this quote as an explicit design constraint, not as an abstract idea: no SaaS at all, no proprietary integration, with the entire rotation expressed in Python data structures simple enough that exporting them to YAML or JSON — the same portable format this lesson's quote recommends — is a single line of code, never a full re-engineering effort. The decision to build the rotation this way, from the start, is the direct application of the lesson this Hacker News quote teaches: portability doesn't get added later, with effort — it gets designed on day one, or it gets paid for later, with real switching costs.
Common mistakes
Interpreting this lesson as "never use a commercial on-call tool" (over-generalizing the quote). What happens: someone concludes that PagerDuty or Opsgenie are, by design, bad decisions for any team. How to spot it: if your takeaway from this lesson is "commercial on-call tools are a mistake." How to fix it: this same guide's Module 4, lesson 7 already named PagerDuty/Opsgenie as the real destination of an alert on a real production team — representative, no $0 tier, but real and widely used. This lesson's point isn't to avoid those tools, it's to not build the entire process inside them with no portability layer of your own — the difference between using a tool and being locked into it.
Assuming Google SRE's 25% limit is just an arbitrary corporate figure, irrelevant to a small team like Andes Cargo (a repeat of the same error already named in Module 1, lesson 7, now with real design consequences). What happens: someone designs a rotation with only two people, without checking whether that means each one is on primary on-call 50% of the time. How to spot it: if your on-call rotation doesn't include enough people for nobody to exceed 25% of time on primary on-call. How to fix it: this module's lesson 7 is going to design Andes Cargo's rotation with exactly the minimum number of people that respects that limit — four people, one week out of every four, is exactly 25%, the ceiling, not a comfortable margin below it.
Treating "well-designed on-call" as a synonym for "reliable system" (confusing response capacity with prevention). What happens: someone, after building lesson 7's rotation, assumes Andes Cargo's reliability is already solved, with nothing else needing improvement. How to spot it: if your reasoning is "we already have on-call, we're already covered." How to fix it: this lesson was explicit in the "What it does NOT solve" section — an on-call rotation is response capacity, not a prevention mechanism. This guide's Module 7 (postmortems, runbooks, action items) is, specifically, the work that reduces how often on-call needs to respond in the first place; without that work, the best rotation in the world only distributes the same volume of interruptions more fairly, without reducing it.
Exercises
Exercise 1 — Explain, without using the word "SaaS," what makes on-call a type of tool with particularly high "switching costs," compared to, say, switching hosting providers for a static website.
See solution
Switching hosting providers for a static site generally requires moving files and updating a DNS record — a contained technical change, with little to no human process intertwined. On-call, in contrast, isn't just a piece of infrastructure: it's an operational human process — who responds, in what order it escalates if they don't answer, how it integrates with every monitoring system that already exists — that, over time, becomes deeply intertwined with the team's real habits. Switching tools isn't just moving data; it's rebuilding escalation chains, reintegrating every alert source, and retraining every person on the new flow — exactly the kind of cost "higher than almost any other category" this lesson's quote describes.
Exercise 2 — A colleague, after reading this lesson, proposes that Andes Cargo should never use any commercial on-call tool, not even in real production with a large team. Using this lesson's common mistakes section, how would you respond?
See solution
This lesson doesn't argue against using commercial on-call tools — this same guide's Module 4, lesson 7 already named them as the real, expected destination of an alert on a real production team. This lesson's argument is more precise: the risk isn't in using a commercial tool, it's in building the entire on-call process with no portability layer of your own underneath it. A team can use PagerDuty in production and, at the same time, keep its schedules and rotation logic in its own portable format (like the oncall/schedule.py lesson 7 builds) that simply syncs to the commercial tool — that way, if it ever switches vendors, the rotation's real logic isn't lost and doesn't need to be rebuilt from scratch.
Exercise 3 — Explain why this lesson claims that a perfect on-call rotation, combined with a badly designed severity matrix, "burns out any team just as fast." What do those two problems have in common?
See solution
Both problems produce the same end result — frequent, unnecessary interruptions to whoever is on call — just through different paths: a badly designed rotation unfairly concentrates interruptions on a few people; a badly calibrated severity matrix, one that fires SEV1 for incidents that are really SEV3 or SEV4 (the error already named in this module's lesson 3), multiplies the total number of interruptions for any person on call, no matter how fair the shift distribution is. A perfect rotation, layered on a severity matrix that calls everything "urgent," still distributes, perfectly fairly, a volume of interruptions that should never have existed in the first place — fixing only one of the two problems leaves the other intact.
Summary and next step
This lesson extended the on-call definition already quoted in Module 1 with the angle of real implementation cost: the full Hacker News quote on why commercial on-call tools have particularly high switching costs, and the recommendation to treat on-call routing as a portable layer, not as a workflow built inside a single vendor with no way out. You named, with the same honesty as the rest of this guide, what a well-designed rotation DOES solve ("who do I call" ambiguity, fair distribution, verifiable compliance with the 25% limit) and what it DOESN'T (the actual number of incidents, the human cost of being interrupted, a bad severity matrix, and vendor lock-in risk).
Before moving on you should be able to: quote jamiemallers' central phrase on switching costs; explain the difference between "using" a commercial on-call tool and "being locked into" it; and name, from memory, the three things an on-call rotation alone never solves.
Lesson 7 applies this lesson directly: oncall/schedule.py, a fixed, deterministic weekly rotation, built from day one with no SaaS at all, on data structures simple enough that exporting them to a portable format — exactly this lesson's quoted recommendation — is trivial.
Resources
- Hacker News — comment by jamiemallers (#47088882) — the full source of the quote on on-call's switching costs.
- This same repository, Module 1, lesson 7 (
07-the-vocabulary-youll-use-all-guide.md) — the definition of on-call and the 25% limit, quoted again in this lesson. src/paths/aws-cloud-ecosystem/VALIDACION.md— the source that identified this quote as the critical market gap motivating this module.- This same repository, Module 4, lesson 7 (
07-hands-on-routing-the-alert.md) — PagerDuty/Opsgenie named as an alert's real destination in production, this lesson's honest counterpoint.