Severity and escalation: who gets woken up

Description

Lesson 4's runbook ends at communicate, and that communication included, almost in passing, the word SEV2. This lesson gives that label real content: severity — an explicit classification of how serious an incident is, that decides something very concrete: who gets notified, with what urgency, and whether that includes waking someone up at three in the morning. Without a severity criterion, a team ends up either waking everyone up for anything, or letting a serious incident go unnoticed until the next morning — both extremes are the same kind of failure: nobody decided, with judgment, who needed to know.

Connection to the module. Severity doesn't change what gets done during the incident — lesson 4's runbook and lesson 5's order still apply the same way —; it changes the urgency and scope of who gets involved. This lesson builds classifySeverity(), the model that decides that with judgment, and sets up lesson 7, where closure communication depends directly on how severe the incident was.

An analogy: emergency room triage

An emergency room doesn't treat patients in the order they arrived — it classifies them first. Someone with a broken finger waits; someone with severe difficulty breathing goes in immediately, no matter who arrived first. That classification — triage — doesn't depend on how much pain each patient perceives, or how loudly they ask for attention: it depends on objective criteria, decided in advance by trained people, about how serious each situation is and how fast it can get worse without intervention.

Classifying a software incident's severity serves exactly that function. Not every broken guardrail is the same: one can affect the entire user base with checkout completely down, and another can affect a small percentage with a degradation that has a workaround available. Treating both with the same urgency — waking up the whole team equally for either, or leaving both for the next day — ignores the information triage is designed to capture.

Worked example: classifySeverity() on three Mercado incidents

// classifySeverity: SEV1 (everyone up, now), SEV2 (on-call + lead), SEV3 (business
// hours, no pager). The % of affected users alone isn't enough -- it also matters
// whether a core function is down and whether a workaround exists.
function classifySeverity({ percentUsersAffected, coreFunctionDown, hasWorkaround }) {
  if (coreFunctionDown && !hasWorkaround) {
    return { severity: 'SEV1', paged: 'on-call + incident commander + leadership, immediately' };
  }
  if (percentUsersAffected >= 0.05 || (coreFunctionDown && hasWorkaround)) {
    return { severity: 'SEV2', paged: 'on-call + team tech lead' };
  }
  return { severity: 'SEV3', paged: 'nobody outside business hours -- handled the next business day' };
}

const incidents = [
  {
    label: 'recommendations, rollout 10% (this guide\'s case)',
    percentUsersAffected: 0.10,
    coreFunctionDown: false, // checkout is STILL working, just slower
    hasWorkaround: true,     // the kill switch shuts off recommendations without touching the rest of the site
  },
  {
    label: 'checkout fully down, no way to process payments',
    percentUsersAffected: 1.00,
    coreFunctionDown: true,
    hasWorkaround: false,
  },
  {
    label: 'a broken icon in the recommendations carousel, cosmetic',
    percentUsersAffected: 0.01,
    coreFunctionDown: false,
    hasWorkaround: true,
  },
];

console.log('=== severity of three Mercado incidents ===\n');
incidents.forEach((i) => {
  const result = classifySeverity(i);
  console.log(i.label);
  console.log('  affected=' + (i.percentUsersAffected * 100) + '%  coreFunctionDown=' + i.coreFunctionDown + '  hasWorkaround=' + i.hasWorkaround);
  console.log('  -> ' + result.severity + '  |  paged: ' + result.paged + '\n');
});

What to expect. Running the file with Node, the output is exactly this:

=== severity of three Mercado incidents ===

recommendations, rollout 10% (this guide's case)
  affected=10%  coreFunctionDown=false  hasWorkaround=true
  -> SEV2  |  paged: on-call + team tech lead

checkout fully down, no way to process payments
  affected=100%  coreFunctionDown=true  hasWorkaround=false
  -> SEV1  |  paged: on-call + incident commander + leadership, immediately

a broken icon in the recommendations carousel, cosmetic
  affected=1%  coreFunctionDown=false  hasWorkaround=true
  -> SEV3  |  paged: nobody outside business hours -- handled the next business day

The three incidents show the three levels, and it's worth noting why each one lands where it does. The recommendations incident — this guide's central case — is SEV2: it affects 10% of the base, a number above the 5% threshold, but checkout keeps working and an immediate workaround exists (the kill switch). It isn't SEV1 because nothing critical is fully down without an alternative. Checkout fully down, with no way to process payments and no workaround at all, is the clear SEV1 example — the combination of "core function down" and "no workaround" triggers the model's first condition, regardless of the exact percentage of users. The broken icon, affecting barely 1% and not touching any core function, is SEV3 — something real, worth fixing, but not something that justifies pulling anyone out of their normal hours.

Why the model doesn't rely only on the percentage of affected users

Notice the model's first condition: coreFunctionDown && !hasWorkaround triggers SEV1 without looking at percentUsersAffected at all. This is intentional, and it's worth understanding why. A low percentage of users affected by something truly critical — for example, 1% of payment transactions silently failing, with no workaround at all — can be far more serious than a high percentage of users seeing a minor degradation with a workaround available, like the recommendations case. If the model only looked at the percentage, it would end up underestimating incidents small in scope but catastrophic in impact, and overestimating incidents large in scope but manageable — exactly the same kind of mistake module 4 already avoided by not trusting a single isolated number without context.

This also explains why recommendations at rollout 10%, despite affecting 25,000 real people, isn't a SEV1: the recommendation system has an immediate, low-risk workaround (the kill switch from lesson 2), and it isn't part of Mercado's core purchasing function — checkout keeps processing payments normally throughout the entire incident. That doesn't mean the incident is minor or doesn't matter: it specifically means it doesn't warrant waking up all of leadership at three in the morning — the on-call person and the team's tech lead are enough to respond with the runbook already built.

Common mistakes

Not defining any severity criteria, and waking up the whole team for any alert (or nobody, for none). What happens: without a model like classifySeverity(), each on-call person decides with their own judgment whether something warrants escalating — some exaggerate the urgency of minor incidents (exhausting the team with constant alerts), and others underestimate serious incidents (leaving them unattended until they get worse). Why it happens: without explicit criteria agreed on in advance, the decision "does this deserve waking someone up?" is left to individual judgment, at a moment — at night, under pressure — where individual judgment is less reliable. How to spot it: if two different on-call people would classify the same incident with different severities, no shared criterion exists — only different intuitions. How to fix it: like this lesson's model, define severity criteria with the team, calmly, before a real incident puts them to the test.

Treating severity as fixed from the moment it's detected, without letting it change. What happens: an incident gets classified as SEV2 at the start and nobody re-evaluates it, even if the situation gets worse — for example, if recommendations's broken guardrail unexpectedly started affecting checkout too. Why it happens: classifying once feels like a completed task, and re-evaluating in the middle of the response can feel like a distraction from the work of mitigating. How to spot it: if an incident lasts a long time or changes in scope, and the severity assigned at the start is never reviewed, that's a sign the process treats classification as a one-time event instead of a state that gets updated. How to fix it: rerun classifySeverity() (or its human equivalent) every time the available information changes significantly during the incident, not just at the start.

"Keep going because the primary metric wins" also applies here: using a positive experiment result to justify a lower severity than warranted. What happens: someone argues that, since recommendations is winning +18.75% on conversion, the latency incident "isn't that serious" and should be downgraded from SEV2 to SEV3. Why it happens: the same bias lesson 1 named about rollbackDecision() reappears here, in another form: a large positive result makes any associated problem feel less urgent than it actually is. How to spot it: if the argument for lowering an incident's severity mentions the experiment's result instead of the model's real variables — percentUsersAffected, coreFunctionDown, hasWorkaround — the argument is mixing two questions that should stay separate. How to fix it: like classifySeverity() in this lesson, severity is calculated exclusively from variables about the incident's own impact — the result of the experiment that caused it isn't one of those variables, and it shouldn't be.

Exercises

Exercise 1 — Change a single variable. If the recommendations incident at rollout 10% had happened at rollout 1% (canary), with only 2,500 users exposed (percentUsersAffected: 0.01) and the rest of the variables unchanged, what severity would classifySeverity() return? Why does that change make sense?

See solution

With percentUsersAffected: 0.01, coreFunctionDown: false, and hasWorkaround: true, neither of the first two conditions is met (coreFunctionDown is false, and 0.01 >= 0.05 is false), so the result would be SEV3 — not SEV2. It makes sense: the same kind of problem, with the same workaround available, affects far fewer people at the canary stage than at rollout 10%. This illustrates why the gradual ramp from modules 2 and 3 doesn't just limit a bug's blast radius — it also naturally limits the severity of the incident that bug can generate if it's caught early, at an early stage.

Exercise 2 — Find the edge case. What severity would classifySeverity() return for an incident with percentUsersAffected: 0.05 exactly (neither more nor less), coreFunctionDown: false, hasWorkaround: true? Check the operator used in the model's condition carefully.

See solution

SEV2. The condition is percentUsersAffected >= 0.05, with a >= that includes the exact value of 0.05 — so an incident affecting exactly 5% of users falls into SEV2, not SEV3. This is a good exercise for noticing something important about any model with thresholds: edge cases (exactly 0.05, neither a bit more nor a bit less) depend on the exact operator used (>= versus >), and it's worth checking explicitly instead of assuming which side of the boundary each category corresponds to.

Exercise 3 — Explain triage without using the word "severity." In two or three sentences, explain to someone new on the team why Mercado doesn't wake up the entire leadership team every time something breaks, and also doesn't wait until the next day to handle any incident. You can use the emergency room analogy.

See solution

One example answer: "We do the same thing an emergency room does with its patients: we don't treat everything with the same urgency, and we don't ignore everything equally until the next day either. We look at how serious each problem is — is it affecting something essential like payments? do we have a fast way to contain it without waiting? — and with that information we decide whether someone needs to be woken up immediately or whether it can safely wait until morning. That avoids two problems: exhausting the team with alerts over minor things, and letting something serious slip by without anyone finding out in time." The central idea: the classification isn't about how annoying a problem feels — it's about how fast it needs intervention before it gets worse.

Summary and next step

In this lesson you built classifySeverity(), the model that classifies an incident as SEV1, SEV2, or SEV3 based on whether a core function is down, whether a workaround exists, and what percentage of users is affected — and decides, with that result, who gets notified and with what urgency. On the real recommendations incident at rollout 10%, the classification was SEV2: serious, with on-call and the tech lead involved, but with no need to wake up all of leadership, because checkout keeps working and the kill switch offers an immediate workaround.

Before moving on you should be able to: explain why the model evaluates coreFunctionDown before the percentage of affected users; identify the mistake of letting a positive experiment result influence the assigned severity; and classify a new incident, given its scope and conditions, into one of the three levels.

You already have four pieces: the decision (rollbackDecision()), the runbook, the correct order (mitigate before diagnosing), and severity. Lesson 7 closes the loop with the piece that's missing: how do you measure, with a concrete number, how fast the response to this incident actually was — and how do you communicate that closure?

Resources

• PagerDuty Incident Response Documentation, "Severity Levels" — response.pagerduty.com/before/severity_levels. The industry's direct reference on how to define severity levels in advance, and why generic definitions need to be adapted to each organization's concrete numbers — exactly what classifySeverity() does with its thresholds. In English.
• Atlassian, "Understanding incident severity levels" — atlassian.com/incident-management/kpis/severity-levels. Describes the SEV1/SEV2/SEV3 scheme with criteria similar to this lesson's: scope of impact, availability of a workaround, and whether a core function is affected. In English.
• Google SRE Book, Chapter 14, "Managing Incidents" — sre.google/sre-book/managing-incidents. Describes Google's incident command system, where severity explicitly determines which roles get activated and with what urgency — the same principle behind classifySeverity()'s paged field. In English.