Module 7: Graceful Degradation and Load Shedding
3. Critical vs non-critical
Overview
In the previous lesson you degraded shipping and didn't touch payments, and you did it as if it were obvious. It isn't —it's the most important design decision of the whole module, and the one that enables it—. Before you can degrade anything, you have to answer one question for each dependency of your system: if this goes down, does the user still get the central value of the operation? If the answer is yes, the dependency is non-critical and its failure can be degraded (deferred, skipped, using a fallback). If the answer is no, it's critical and its failure must be able to veto the operation. All the module's degradation rests on this classification; doing it wrong breaks it in both directions.
The reason this lesson exists separately is that the classification is where the most expensive mistakes are made, and they're silent mistakes —they don't break a test, they don't throw an exception; they only manifest in production as lost money or avoidable outages—. Classifying payments as non-critical (and degrading it) makes the checkout "complete" orders you never charged: you give away product. Classifying reviews as critical (and letting its failure knock down the page) makes a cosmetic service cause outages of your whole catalog: you lose sales for nothing. Both mistakes are easy to make because the classification isn't a technical property of the dependency —it's a property of the business—, and engineers tend to treat all network calls as equal. They're not: some carry the central value and others adorn it.
Connection with the module: this lesson is the hinge. Lessons 1 and 2 assumed the classification; lessons 4, 5 and 6 use it (a fallback, a deferral and a toggle only make sense over a non-critical dependency). Here you learn to do it: the question that classifies, the table of Mercado's dependencies, the two classification mistakes measured, and the nuance that "critical" is neither binary nor permanent (there are degrees, and they depend on the context). Leave here knowing how to look at your own system and say, for each dependency, "this can knock me down, this can't" —and why—.
The analogy: what's essential for the plane to fly
A plane has hundreds of systems, and aviation engineers classify them with exactly the same question you're going to use: is this system essential for the flight?
The engines are critical. Without thrust there's no flight; their failure is a legitimate veto on "flying." That's why they're designed with redundancy (two, four engines) and don't "degrade": an engine can't fail soft, it has to work. The flight control (ailerons, rudder) is critical. The cabin pressurization is critical at high altitude.
The in-flight entertainment —the little seatback screens— is non-critical. If the video system goes down mid-flight, does the plane land anyway? Yes, perfectly. No one diverts a flight because the movies stopped working; at most the flight attendant announces "the entertainment isn't available on this flight" —an honest degradation— and everyone reaches their destination. The galley coffee is non-critical. Your seat's reading light is non-critical.
The question that separates the two lists isn't "how complex is the system?" nor "how much did it cost?" —the entertainment screens are super expensive and complex, and even so they're expendable—. The question is "does the plane fulfill its essential function (getting you safely to your destination) without this?". That's exactly the question you're going to ask each dependency of your checkout. And notice the design asymmetry: the critical systems carry redundancy and never turn off; the non-critical ones turn off without drama when needed. Your system should do the same: armor the critical (payments) to exhaustion, and degrade the non-critical (shipping, reviews) without guilt.
The question that classifies, and Mercado's table
The operational question, applied to each dependency, is a single one:
If this dependency is down, does the user still get the central value of the operation, even in a reduced version?
- If yes → non-critical. It degrades (fallback, defer or skip). Its failure shouldn't knock down the operation.
- If no → critical. Its failure must be able to veto the operation. It's armored to the max, not degraded.
Let's apply it to Mercado's checkout, whose central value is "the buyer pays and is left with a confirmed order":
Dependency Does the buyer get their paid order without this? Classification
----------- ----------------------------------------------- --------------
payments NO. No charge, no order. It's the central value. CRITICAL
catalog NO/partial. Without the product or price there's CRITICAL
nothing to buy; but non-essential fields (product (with nuances)
reviews) can be skipped.
shipping YES. The shipment can be created later; the buyer NON-CRITICAL
already paid and has their confirmed order.
reviews YES. Reviews adorn; they aren't the purchase. NON-CRITICAL
recommendations YES. "You might also like" is pure extra. NON-CRITICAL
Three observations about this table, because each one is a trap to see.
payments is critical, period. There's no reduced version of "charging." You can't "charge a little" nor "charge later without risking never charging." It's the heart of the transaction, and its failure is the only veto the degraded checkout respects (that's why the checkout reached 99.1% and not 100%: the ~1% are payments' legitimate failures). Armor it with everything from modules 2-6, but don't degrade it.
catalog is critical with nuances —and this shows that the classification isn't always binary—. Without knowing what product and at what price, there's no purchase: in that sense it's critical. But catalog serves many fields, and not all are equally critical: the price is critical (charging an old price is a problem), but the long description or the product review list are skippable. So catalog is an example of a partially degradable dependency: some of its fields veto, others can be served cached or empty. The fine classification isn't always "the whole dependency," sometimes it's "field by field."
shipping is non-critical, and that's why you could defer it. The buyer already paid and has their confirmation; whether the shipping label is generated now or in five minutes doesn't change their experience. This is the underlying reason the degradation worked in lesson 2 —it wasn't a trick, it was the reflection of a business truth: the shipment is deferrable—.
Worked example: what happens if you classify wrong
The classification isn't philosophy; it has measurable consequences. We're going to measure the most expensive mistake: degrading a critical dependency. We take the degraded checkout from lesson 2 and, by a "classification error," treat payments as non-critical —if the charge fails, we degrade: we complete the order anyway—. With shipping down (as before) and payments failing at 1%:
def checkout_misclassified(rng, payments, shipping, deferred_queue):
"""MISTAKE: payments treated as non-critical (its failure is degraded)."""
pay_ok = payments.call(rng)
if not pay_ok:
deferred_queue.append("retry_charge") # <-- MISTAKE: degrade the charge
# the order "completes" without a confirmed charge
ship_ok = shipping.call(rng)
if not ship_ok:
deferred_queue.append("create_shipment")
return "completed" # completes ALWAYS, even without charging
What to expect. Compared to the well-classified checkout, over the same 2000 checkouts and the same seed:
CLASSIFICATION OF payments · shipping down · 2000 checkouts
(a) payments CRITICAL (correct)
completed orders : 1981 / 2000 (99.1%)
orders completed WITHOUT charge : 0
-> the ~0.9% "lost" are charges that failed: correct veto
(b) payments NON-CRITICAL (misclassified)
completed orders : 2000 / 2000 (100.0%)
orders completed WITHOUT charge : 19
-> 19 "successful" orders you never charged: product given away
This is the mistake that doesn't throw an exception and that's why it's so dangerous. The misclassified version shows 100% success —better than the "correct" 99.1%!—, and there's the trap: the number goes up, so it seems an improvement. But 19 of those "successes" are orders that completed without a confirmed charge: product Mercado is going to ship without having received the money. The 100% isn't better than the 99.1%; it's 99.1% of real sales plus 0.9% of giveaways. The correct classification sacrifices that 0.9% on purpose, because a failed charge must knock down the order —that veto is the only defense against giving away product—.
The moral discomforts the instinct: a higher success number can mean a worse classification. Over-degrading inflates the "completed operations" metric while emptying the cash register. That's why the classification isn't validated by looking at the success_rate —which deceives—; it's validated by asking, for each "successful" operation, whether it really delivered the central value. An order without a charge didn't deliver it, no matter how much the counter says "completed."
The mistake in the other direction —classifying reviews as critical— is less venomous but also costs: if you let reviews' failure (a cosmetic service that goes down often) knock down the product page, you turn every hiccup of a secondary system into an outage of your catalog. You lose real sales to "protect" reviews that don't stop anyone from buying. You measure it in lesson 4 (the product page goes from 0% to 99.6% precisely by not treating reviews as critical).
Deep dive: criticality is neither binary nor permanent
Two nuances that separate a naive classification from a mature one.
It's not binary: there are degrees. Between "absolutely critical" (payments) and "pure adornment" (recommendations) there's a spectrum. A useful way to grade it is the degradation levels: instead of "works or doesn't work," you define intermediate states. For Mercado's product page, for example: level 0 (everything: product + reviews + recommendations + live stock), level 1 (product + cached reviews, no recommendations), level 2 (product + price, no reviews or recommendations), level 3 (just "product temporarily unavailable"). Each dependency that goes down drops you a level, it doesn't turn you off. Designing these levels before the incident is what turns an outage into an ordered degradation instead of an improvised collapse.
It's not permanent: it depends on the context. The same dependency can be critical in one flow and non-critical in another. shipping is non-critical in the checkout (the shipment is deferred), but it would be critical in a "track my package" flow (there shipping is the central value; without it there's nothing to show). catalog is critical when buying, but it could degrade to cached data when just browsing. And the criticality changes with the business: in an online pharmacy, "check drug interactions" is critical in a way that "recommended products" in a clothing marketplace isn't. That's why the classification isn't inherited from a generic pattern catalog: it's done for your system, in your flow, with your definition of central value. The question is universal; the answer is yours.
A third nuance, operational: the classification should be written and visible, not in the head of whoever coded the checkout. A good artifact is a dependency map that, for each one, notes: criticality, degradation strategy (fallback / defer / skip), and what's shown to the user when it goes down. That map is what lets whoever's on call, at 3 a.m. during an incident, know that a downed shipping isn't an emergency (it only degrades) while a downed payments is (it blocks sales).
Common mistakes
Treating all network calls as equal. What happens: all the dependencies are wrapped in the same timeout, the same retry, the same error handling, without distinguishing criticality. Why it happens: from the code, payments.call() and reviews.call() look identical —they're two HTTP calls—. How to spot it: if your checkout treats reviews' failure the same as payments', either you can't degrade (everything vetoes) or you over-degrade (nothing vetoes). How to fix it: criticality is a label you put according to the business, not a property the protocol gives you. Note it explicitly per dependency, and let that label decide the failure handling: critical → vetoes, non-critical → degrades.
Validating the classification by the success_rate. What happens: "the checkout is at 100%" is looked at and it's concluded "we classified well." Why it happens: a high number feels like health. How to spot it: this lesson's example —the incorrect classification gave 100%, higher than the correct one (99.1%)—. How to fix it: the success_rate doesn't distinguish a real sale from a giveaway. Validate the classification by asking, for each "success," whether it delivered the central value and charged what it should. An honest metric isn't "completed orders" but "completed and charged orders." If those two figures differ, you degraded something critical.
Freezing the classification. What happens: it's classified once, at design time, and never revisited. Why it happens: it seems a startup decision, made forever. How to spot it: if your criticality map is two years old and the system changed (new flows, new dependencies, new business model), it's out of date. How to fix it: criticality depends on the flow and the business, and both evolve. A dependency that was adornment can become central (if Mercado adds "buy with verified reviews," reviews rises in rank). Revisit the classification when the flow changes, not just when something breaks.
Exercises
Exercise 1 — Classify a new flow. Mercado adds an "order detail" page that the buyer sees after buying, with this data: the payment status, the shipping status (tracking), the purchased products (from catalog), and "other buyers also took" (from recommendations). For each one, say whether it's critical or non-critical for this page and why. Does any dependency change criticality relative to the checkout?
See solution
- Payment status: critical. The central value of "order detail" is knowing the status of your purchase; without it the page doesn't fulfill its function.
- Shipping status / tracking (
shipping): critical here —and this is the key change—. In the checkout,shippingwas non-critical (the shipment was deferred). But in "track my order," the shipping status is a good part of the central value: the user entered precisely to see where their package is. The same dependency, opposite criticality depending on the flow. - Purchased products (
catalog): critical/partial. You need to show what was bought; but you could serve cached name and image (they don't need to be fresh, the order was already placed). Degradable to cached data. - "Others also took" (
recommendations): non-critical. Pure extra; skipped if it goes down.
The lesson: shipping went from non-critical (checkout) to critical (tracking). Criticality is not a property of the dependency, it's a property of the dependency in a flow. The same service is armored or degraded depending on what central value it's serving.
Exercise 2 — The deceiving success_rate. A team proudly reports that they raised the checkout from 98.5% to 99.8% "by improving error handling." Investigating, you discover that what they did was start degrading payments' failure (complete the order and retry the charge later, in a queue). (a) Why did the number go up? (b) Is it an improvement? (c) What metric would have revealed the problem, and what would you have recommended instead?
See solution
(a) It went up because, by degrading payments' failure, those checkouts that used to count as "failed" (rejected charge) now count as "completed" (order created, charge pending in queue). Moving legitimate failures to "success with deferred charge" inflates the numerator.
(b) No —or at least, it's a risky business decision disguised as a technical improvement—. Deferring the charge means Mercado confirms and potentially ships orders whose charge hasn't yet succeeded; if those deferred charges fail definitively (card with no funds, fraud), it's product given away or a collections mess. payments is critical precisely because its failure must be able to veto. There may be business cases where a deferred charge makes sense (subscriptions with retry, a trusted buyer), but that's a deliberate financial-risk decision, not "better error handling," and it needs controls (credit limit, verification) that a simple append to the queue doesn't have.
(c) The metric that reveals it is "completed and charged orders" (or its inverse: "orders completed without a confirmed charge"). If "completed" went up but "completed and charged" didn't, the difference is giveaways/risk. I would have recommended: keep payments as critical (its failure vetoes), and to really raise the number, improve payments' reliability (timeout + retry with backoff + breaker from modules 2-6) or give the buyer the option of another payment method —not reclassify the failure as success—.
Exercise 3 — Design the degradation levels. For Mercado's product page —which depends on catalog (product + price + stock), reviews and recommendations— design 3 or 4 explicit degradation levels, from "everything works" to "minimum viable." For each level, say which downed dependency(ies) trigger it and what the user sees. What's the minimum level below which the page no longer makes sense to show?
See solution
A reasonable design (the exact numbers are judgment; what matters is that they're thought out in advance):
- Level 0 — complete. Everything healthy. The user sees: product + price + live stock + reviews + recommendations. Trigger: nothing down.
- Level 1 — no recommendations.
recommendationsdown. The user sees everything except the "you might also like" section (skipped silently; no one misses it). Trigger:recommendationsdown. - Level 2 — no reviews (or cached reviews).
reviewsdown. The user sees product + price + stock, and in place of reviews: "reviews unavailable" or the cached summary. They can still buy. Trigger:reviewsdown. - Level 3 — only the essential to buy.
reviews+recommendationsdown,catalogdegraded to cached data (name, image, verified price). The user sees the minimum to decide the purchase. Trigger: several non-critical down + partial catalog. - Floor — don't show. If
catalogcan't give reliable price and stock, the page no longer makes sense: showing a product without a price or whose stock you can't verify leads to charging errors or selling what isn't there. Here you do show "product temporarily unavailable" —which is an honest degradation, not a hard failure by carelessness—.
The floor is set by the verifiable price and stock, because they're the catalog fields that are indeed critical (charging wrong or selling nonexistent are real harms). Everything else degrades above that floor. What's valuable about the exercise isn't the exact levels, but having defined them before the incident: during the outage you no longer improvise, you drop a level.
Summary and next step
In this lesson you learned the decision that enables all degradation: classify each dependency as critical (its failure must veto the operation; it's armored, not degraded) or non-critical (its failure degrades; fallback, defer or skip), with a single question —"does the user get the central value without this?"—. You saw Mercado's table (payments and catalog critical, shipping/reviews/recommendations non-critical), measured the most expensive mistake (degrading payments gives 100% "success" but 19 orders without a charge: product given away), and saw the nuances that separate a mature classification from a naive one: criticality is neither binary (there are degradation levels) nor permanent (it depends on the flow and the business).
Before moving on you should be able to: classify a dependency with the central-value question; explain why a higher success_rate can hide a worse classification; and design degradation levels for a flow.
You already know what you can degrade (the non-critical). The next three lessons are how. Lesson 4 starts with the most common form: the fallback value —when a non-critical dependency goes down, what do you show in its place?—. You'll see the fallback hierarchy (cached, default, skip, "unavailable"), measure Mercado's product page go from 0% to 99.6% with reviews and recommendations down, and learn the dangerous trade-off every fallback hides: serving an old value that's sometimes harmless and sometimes a bomb.
Resources
- Michael T. Nygard, Release It!, 2nd ed. (Pragmatic Bookshelf, 2018) — graceful degradation starts by deciding what's expendable; the stability patterns assume this classification as step zero. In English.
- Betsy Beyer et al. (eds.), Site Reliability Engineering (O'Reilly, 2016), chapter 22, "Addressing Cascading Failures" — sre.google/sre-book/addressing-cascading-failures. Discusses the "degraded service levels" and why designing them in advance avoids the collapse. Free and in English.
- Google SRE, "Graceful degradation" in The Site Reliability Workbook, chapter on Managing Load — sre.google/workbook/managing-load. Real examples of classifying responses as critical vs expendable to degrade under stress. Free and in English.
- Sam Newman, Building Microservices, 2nd ed. (O'Reilly, 2021), section on resilience — deals explicitly with how to decide which functionality is essential and which can be degraded when a downstream dependency fails. In English.