Module 6: Operating The Claude Code Incident

1. Introduction: operating, not narrating again

Description

If you got here following the whole ecosystem, this is already the fourth time you've encountered this same incident: a Claude Code agent ran terraform destroy -auto-approve against DataTalks.Club's production infrastructure, with a stale state as its reference, and erased 2.5 years of data — the courses_answer table, with 1,943,200 rows, among them. terraform-and-iac-guide narrated it as a lesson about reading the full plan. cloud-security-and-guardrails-guide narrated it as a lesson about automated controls (conftest, prevent_destroy) that don't depend on human attention. This same guide's Module 1 picked it back up a third time, with the new error budget vocabulary, and calculated that the ~24 hours of recovery consumed 33.3 times the full monthly budget of a hypothetical 99.9% SLO.

This module doesn't narrate it a fifth time. It operates it. The difference is complete, not a nuance: lessons 2 through 8 of this module don't retell what happened — they take the complete framework Module 5 just finished (INCIDENT-RESPONSE-PLAN.md: the five-stage lifecycle, the burn-rate-tied severity matrix, the three roles, the on-call rotation) and run it, step by step, against this real incident, exactly as you would run it against any new incident Andes Cargo had to resolve. The result isn't a more complete narrative — it's a series of executable documents: a TIMELINE.md with relative timestamps, a severity classification with the arithmetic in plain sight, a declaration message with roles assigned, a mitigation decision tree mapped against the lifecycle, and a recalculation of the error budget with the real calculator, not a one-off script.

Connection to the module

Everything this module needs already exists, built in the five previous modules: SLO.md (Module 2) defines the error budget lesson 7 measures the incident against; ALERTING-POLICY.md (Module 4) defines the burn rate thresholds the severity matrix already uses; INCIDENT-RESPONSE-PLAN.md (Module 5) defines the lifecycle, the severity matrix, the roles, and the on-call rotation this module's lessons 2 through 6 apply, without modifying a single figure. This module declares no new HCL resource, runs no new LocalStack command, builds no piece of infrastructure — it builds operational documents, the same kind of artifact a real incident at any company produces while it's happening: a timeline, a declaration, a record of mitigation decisions. Module 7 takes the TIMELINE.md this module finishes and writes, on top of it, the complete blameless postmortem — the "why it happened" question this module deliberately doesn't answer yet.

The analogy: a fire drill, with a real fire already documented

A fire department trains with drills — an empty building, artificial smoke, a situation designed to practice the protocol without real risk. But there's a second kind of training, even more valuable, that almost no fire department can afford: a real fire, already happened, already documented with complete forensic precision — where it started, how fast it spread, what decisions each person involved made, how long it took to contain — that a new team can run through its own current protocol, step by step, as if it were happening right now. There's no real risk — the fire ended months ago — but none of an invented drill's artificial simplifications either: the facts are real, verified, with a primary source anyone can check.

That is exactly what this module does. The Claude Code incident already happened, is already fully documented — by the affected person themselves, by an independent AI incident database, by verified technical coverage — and the figures admit no ambiguity: 2.5 years of data, 1,943,200 rows, exactly 24 hours of recovery. This module takes that real fire and runs it against Andes Cargo's current protocol — INCIDENT-RESPONSE-PLAN.md, finished in Module 5 — exactly as you'd run any new incident: it gets detected, declared, assigned a severity, assigned roles, mitigated, resolved. The only difference from a new incident is one thing: you already know how it ends. That doesn't reduce the exercise's value — it lets you focus completely on whether the protocol itself works, without the added uncertainty of an unknown outcome.

   TWO KINDS OF TRAINING -- AND WHY THIS MODULE USES THE SECOND

   INVENTED DRILL                          REAL FIRE, ALREADY DOCUMENTED
   ────────────────────                    ──────────────────────────────
   Hand-designed scenario,                 The Claude Code destroy incident,
   with no real supporting data             with a verified primary source
        │                                        │
        ▼                                        ▼
   Practices the protocol under             Practices the EXACT protocol
   controlled conditions,                   against facts that already
   but without the real texture             happened, with the full texture
   of a real event                          of a real case (and its gray areas)
        │                                        │
        ▼                                        ▼
   Useful for training basic                Useful for VALIDATING that the
   reflexes                                 framework you already built
                                             (Module 5) truly resolves a real case

What you're going to build in this module

Eight lessons, each a concrete step of the same journey — from reconstructing the facts to the final portfolio document Module 7 is going to cite:

#	Lesson	What it produces
1	This introduction	The module's complete map, and why "operating" is different from "narrating"
2	Reconstructing the exact timeline	`incidents/2026-02-26-claude-code-destroy/TIMELINE.md` — first version, verified facts, relative marks (`T+0`)
3	Classifying the severity	SEV1, with the arithmetic (or lack of it) that justifies it
4	Declaring the incident	The declaration message, with roles assigned from `INCIDENT-RESPONSE-PLAN.md`
5	Internal vs. external communication	What was really communicated publicly, versus what should have been communicated internally first
6	The mitigation decision tree	The real recovery mechanism, mapped against Module 5's five stages
7	This incident's error budget, with the full vocabulary	The same 33.3x from Module 1, now with a real `error_budget_calculator.py`
8	Project: the final `TIMELINE.md`	The complete document — timeline + severity + roles + mitigation, integrated

Each lesson runs a different stage of the five-step lifecycle Module 5 already quoted from Google SRE (detection → declaration → response → mitigation → resolution), in the same order that cycle really occurs:

   MODULE 5'S LIFECYCLE, APPLIED TO THE REAL INCIDENT

   DETECTION           DECLARATION         RESPONSE            MITIGATION         RESOLUTION
   (Lesson 2,          (Lesson 4)          (Lessons 4-5,       (Lesson 6)         (Lessons 2, 6)
    part of the                             roles + comms
    timeline)                               cadence)

   T+0: destroy         T+~5min:            Ana (IC+CL),        The real          T+24h00m:
   runs -- the whole    "this is SEV1,      Bruno (OL)          decision tree:     snapshot
   system stops         declared"           coordinate the      self-service       restored,
   responding                                response            fails -->         1,943,200
                                                                 AWS support -->    rows intact
                                                                 internal snapshot

Why this module does NOT narrate the incident again (and what it does instead)

It's worth being precise about the difference, because it's easy to confuse "operating" with "telling the story in more detail." It isn't that. The three sibling guides that already used this case — terraform-and-iac-guide, cicd-and-gitops-on-aws-guide, cloud-security-and-guardrails-guide — and this guide's own Module 1 already established, with complete precision, what happened: who did it, with what command, against what infrastructure, with what exact result. That work isn't repeated here.

What this module adds is a completely different question: if Andes Cargo had to operate this exact incident today, with the framework Module 5 just finished, what would it produce, step by step? It's not a rhetorical question — every lesson in this module produces the real document that question demands: a formal timeline, a severity classification with INCIDENT-RESPONSE-PLAN.md's exact matrix, a declaration message with roles assigned from Module 5's real rotation, a decision tree that documents every branch of the real 24-hour response. It's the difference between reading about a fire and running, with the real stopwatch running, your own fire department's complete protocol against that fire's facts.

An honesty that needs to be stated clearly from this first lesson: the Claude Code incident happened to DataTalks.Club, not to Andes Cargo. Andes Cargo is this ecosystem's fictional project; DataTalks.Club is a real platform, with real data, affected by a real incident. This module doesn't pretend the incident "happened to" Andes Cargo — it takes the verified facts of a real, external case and runs them through Andes Cargo's operational machinery as a deliberate exercise, exactly the same kind of exercise any real SRE team does when it studies another company's public postmortem to stress-test its own protocol. Every lesson in this module, wherever it matters, precisely marks this boundary — what is a verified fact of the real incident, and what is a design decision of this exercise (like which person from Andes Cargo's rotation took on each role).

Common mistakes

Expecting this module to reveal a new fact about the incident that the three sibling guides didn't mention (misdirected expectation). What happens: someone arrives at this module looking for a different figure, an alternative root cause, or some angle that contradicts what was already established in terraform-and-iac-guide or cloud-security-and-guardrails-guide. How to spot it: if your reading of this module looks for "what's new about what happened" instead of "what's new about how it's operated." How to fix it: the central figures — 2.5 years, 1,943,200 rows, exactly 24 hours, February 26, 2026 — are the same across all four guides, because they're facts verified against the same primary source, not a narrative that varies. What's new in this module is never an additional fact about the incident — it's the complete operational process that runs on top of those same facts.

Treating this module's exercise as if the incident literally happened to Andes Cargo (losing the boundary between the real case and the exercise). What happens: someone, in a technical conversation or an interview, describes the Claude Code incident as "something that happened to Andes Cargo," instead of to DataTalks.Club. How to spot it: if your summary of the module says "Andes Cargo suffered an incident where..." instead of "we operated, with Andes Cargo's framework, a real incident that happened to DataTalks.Club." How to fix it: this lesson made it explicit — Andes Cargo is this ecosystem's fictional project; DataTalks.Club is the real affected company. The exercise's value doesn't depend on the incident having happened to Andes Cargo, it depends on the facts being real and verifiable, and on the framework applied to them (INCIDENT-RESPONSE-PLAN.md) also being real.

Assuming "operating" means rewriting the outcome, or imagining how Andes Cargo would have avoided it (confusing this module with the prevention one). What happens: someone starts designing new preventive controls — an additional prevent_destroy, a different conftest policy — as if that were this module's job. How to spot it: if any lesson's deliverable in this module is a new security rule, instead of an incident response document. How to fix it: prevention was already handled, in another guide — cloud-security-and-guardrails-guide, with conftest and prevent_destroy — and this guide, since RELIABILITY-CHARTER.md in Module 1, already established that an error budget "is a measurement mechanism, never a preventive control." This module assumes prevention already failed — the destroy already ran — and builds response discipline, not prevention: what gets done, in what order, with what roles, once something has already broken.

Exercises

Exercise 1 — In your own words, explain the difference between "narrating" and "operating" an incident, using this module's concrete example. What would a narration of the Claude Code incident produce that an operation of the same incident wouldn't, and vice versa?

See solution

A narration produces a complete, precise story of what happened — the technical detail of how the stale state misled the agent, the exact sequence of commands, the damage figures — optimized for whoever reads it to understand the incident's magnitude and cause. An operation, in contrast, doesn't need to repeat that story in more detail (it already exists, in three sibling guides) — it produces the documents a real team would generate while responding to the incident: a timeline with timestamps, a declaration message with roles assigned, a record of mitigation decisions in the order they were made. The narration answers "what happened?"; the operation answers "what would a team with a real protocol do, step by step, if it had to respond to this?" This module, deliberately, doesn't repeat the narration — it assumes you already know it, from the three sibling guides and this guide's Module 1, and focuses entirely on the second question.

Exercise 2 — Explain why the drill-with-a-real-fire analogy, instead of an invented drill, is more precise for describing this module than a "practicing with a generic case study" analogy. What specific property of the Claude Code incident makes this analogy work better than a hypothetical example?

See solution

A generic or invented case study lets you adjust the facts at will so the exercise "comes out well" — any uncomfortable detail can be simplified, any gray area omitted, and the result is, almost by design, cleaner than any real incident ever is. The Claude Code incident, in contrast, has the full texture of a real event: a human decision to continue despite an explicit warning from the agent itself, a sequence of events that didn't follow the most "educational" possible script, and an outcome that depended on a recovery mechanism (an internal snapshot not listed in the console) that no invented case would include by accident. Running Andes Cargo's protocol against real facts, with all their texture, is a far more honest test of whether that protocol works than running it against a case designed to work out well.

Exercise 3 — RELIABILITY-CHARTER.md (Module 1) established that an error budget "is a measurement mechanism, never a preventive control." Explain why that same principle is the reason this module doesn't try to redesign any preventive safeguard.

See solution

If an error budget measures the damage after it happened, instead of preventing it, then the whole incident response framework this module operates — lifecycle, severity, roles, mitigation — inherits the same nature: it exists to respond well after something has already failed, not to keep it from failing in the first place. Redesigning a preventive safeguard (a new prevent_destroy, an additional conftest policy) would be legitimate work, but it belongs to a different discipline — the one cloud-security-and-guardrails-guide already built — and mixing it into this module would dilute the specific question this module answers: given that the destroy already ran, with no possibility of undoing it with a safeguard, what does a well-prepared team do in the following 24 hours? That's a response question, not a prevention one, and this module holds to it without drifting.

Summary and next step

This lesson established the whole module's central difference: operating an already-known incident means running INCIDENT-RESPONSE-PLAN.md's complete framework (Module 5) against its verified facts, producing real operational documents — not repeating, a fifth time, the narrative three sibling guides and this guide's Module 1 already built. You saw the complete map of the eight lessons, each a step of Google SRE's five-stage lifecycle applied, in order, to the real Claude Code incident; and the honesty holding the whole module together: the incident happened to DataTalks.Club, not to Andes Cargo, and this module is a deliberate exercise in running real facts through a real framework, not a fiction that it happened to a fictional project.

Before moving on you should be able to: explain, in your own words, the difference between narrating and operating an incident; name the five lifecycle stages this module is going to walk through, in order; and explain why this module doesn't redesign any preventive safeguard.

Lesson 2 starts the journey with this module's first real document: TIMELINE.md, reconstructed with the primary source's verified facts, with relative timestamps — never a real clock — exactly wherever the public record doesn't offer minute-level precision.

Resources

Alexey Grigorev — How I Dropped Our Production Database — this whole module's primary source, already cited in this guide's Module 1.
Google SRE — Incident Management Guide — the five-stage lifecycle this module applies, already quoted in full in Module 5.
This same repository, Module 5, lesson 8 (08-project-andes-cargos-incident-response-plan.md) — INCIDENT-RESPONSE-PLAN.md, the complete framework this module's lessons 2-6 are going to operate without modifying.
This same repository, Module 1, lessons 5-6 — the incident's original narration and the first manual error budget calculation, this module's foundation.

← Previous8. Project: Andes Cargo's `INCIDENT-RESPONSE-PLAN.md`Next →2. Hands-on: reconstructing the exact timeline