Module 6: Operating The Claude Code Incident

8. Project: Andes Cargo's final `TIMELINE.md` for this case

Next →

Description

Six lessons built each piece separately: the facts in order (lesson 2), the severity with its exact justification (lesson 3), the declaration with roles assigned (lesson 4), the communication cadence (lesson 5), the mitigation decision tree (lesson 6), and the error budget recalculated with the real calculator (lesson 7). This final project doesn't calculate or investigate anything new — it brings the five technical pieces together into the final version of incidents/2026-02-26-claude-code-destroy/TIMELINE.md: the complete document Module 7 is going to cite directly, without reconstructing any fact from scratch, when writing the blameless postmortem.

Connection to the module

This document follows the same disciplined-assembly principle SLO.md (Module 2) and INCIDENT-RESPONSE-PLAN.md (Module 5) already established: no number or figure in this document is calculated for the first time here — every one was already verified, cited, or calculated in an earlier lesson of this module. This lesson's only genuine novelty is the discipline of integrating them without contradictions, in the same document, in the order a new reader — someone who never saw the six previous lessons — would need to find them.

Step 1 — Why this document cites, instead of repeating, lesson 5

Notice a deliberate design decision: this document does not include lesson 5's complete communication cadence table (the seven internal updates, from T+~5min to T+24h00m). The reason is the same one INCIDENT-RESPONSE-PLAN.md already used when citing SLO.md instead of repeating its content: that level of narrative detail belongs to the lesson that built it, and this document — meant as the final portfolio piece, and as Module 7's direct input for the postmortem — needs to be dense with verifiable facts, not a complete repetition of every lesson. This document's "Roles" section does summarize, in one line, the cadence (from lesson 5), without reproducing the full table.

Step 2 — The complete document

In incidents/2026-02-26-claude-code-destroy/, replace lesson 2's version with the final one:

# TIMELINE.md — 2026-02-26 Claude Code `destroy` Incident

**Status:** Final · **Governs:** input to Module 7's postmortem
**Real incident, external:** happened to DataTalks.Club, not to Andes Cargo. Operated here as a
drill against `INCIDENT-RESPONSE-PLAN.md` (Module 5) — see Module 6, lesson 1.
**Sources:** Alexey Grigorev, "How I Dropped Our Production Database" (alexeyondata.substack.com);
Hacker News #47278720; incidentdatabase.ai #1424; Tom's Hardware.

## A note on timestamps

All timestamps in this document are relative to `T+0`, the moment `terraform destroy
-auto-approve` executed. The primary source gives only approximate clock times; this document
never converts those into exact clock timestamps. Offsets marked `~` are inferred from the
sequence the source describes, not measured to the minute. The two precise offsets in this
document, both stated directly by the primary source: `T+0` itself (the destroy command), and
`T+24h00m` (the restoration — "Exactly 24 hours after the database had been deleted").

## Timeline

| Offset | Event |
|---|---|
| T-~1h | Deployment work begins: migrating a static site into the existing Terraform project, without the correct state file |
| T-~15min | `terraform plan` shows resources marked for creation, not modification — first anomaly noticed |
| T-~10min | Agent explains the loaded state file made it believe "nothing existed"; `apply` stopped, but some resources already partially created |
| T-~5min | Agent proposes `terraform destroy` as the cleanest way to reverse what it had just created |
| T-~1min | Human does not stop the agent |
| **T+0** | **`terraform destroy -auto-approve` executes.** VPC, ECS cluster, load balancers, bastion host, RDS database, and all automated snapshots — destroyed. `courses_answer` (1,943,200 rows, 2.5 years of data) gone. |
| T+0 to T+~5min | Total outage, immediately and unambiguously visible |
| T+~5min | **Incident declared. SEV1.** Roles assigned: Ana (Incident Commander + Communications Lead), Bruno (Operations Lead) |
| T+~1h | Support ticket opened with AWS; upgrade to Business Support tier (~+10% monthly cost) |
| T+~1h30min | AWS confirms a snapshot exists on their end — not visible in the customer console |
| T+~2h to T+~3h | Phone call with AWS support; internal escalation for manual restoration |
| **T+24h00m** | **AWS restores the snapshot.** `courses_answer` restored intact, 1,943,200 rows confirmed. Incident resolved. |
| (8 days after T+0) | Grigorev publishes the full incident writeup publicly |

## Severity

**SEV1** — not by a burn-rate threshold (Module 6, lesson 3: burn rate is mathematically
undefined here, `0/0` — no infrastructure remains to generate measurable traffic) but by
`INCIDENT-RESPONSE-PLAN.md`'s independent SEV1 criterion: unrecoverable data loss without
external support intervention. Verified against four conditions (Module 6, lesson 3, Step 3):
real data loss (yes), recoverable with the team's own tools (no), required external support (yes),
system still responding to other requests (not applicable — nothing was responding at all, a case
more severe than the matrix's minimum bar for this tier).

## Roles

Assigned from `oncall/schedule.py`'s Week 1 rotation (2026-03-02, Ana primary / Bruno secondary),
applied to this incident as a deliberate exercise — the real incident predates the plan's
effective date by six days; see Module 6, lesson 4, Step 1 for the full honesty note.

- **Incident Commander + Communications Lead: Ana** — coordinates the response, owns this
  document, issues updates on a cadence (Module 6, lesson 5): every 15-30 minutes early on,
  spacing out to hours once escalated to AWS and no longer generating new information every few
  minutes.
- **Operations Lead: Bruno** — the only role that would have executed mitigation actions, had any
  been available under the team's own control.

## Mitigation decision tree

Two self-service paths checked and exhausted within minutes of `T+0`: reverting via Terraform
state (not applicable — the state itself was the cause, and the infrastructure it described no
longer existed) and locating an automated RDS snapshot in the account's own console (none —
automated snapshots were destroyed alongside the database). With both own-account paths
exhausted, the incident moved to escalation: AWS Business Support (`T+~1h`), confirmation of an
internal AWS-side snapshot not listed in the console (`T+~1h30min` — the single node in this tree
with genuine uncertainty until AWS responded), phone escalation for manual restoration (`T+~2h`
to `T+~3h`), and a wait outside the team's direct control until `T+24h00m`. Full branch-by-branch
detail: Module 6, lesson 6.

## Error budget

Measured against `SLO.md`'s real SLO (99.9% monthly, 43.2-minute budget) using
`error_budget_minutes()` from `scripts/error_budget_calculator.py` (Module 6, lesson 7 — same
function, same formula the rest of this guide uses, not a one-off calculation):

- **1,440 minutes consumed** (the full 24-hour incident) against a 43.2-minute monthly budget.
- **3,333.3% of the monthly budget consumed** — a deficit of -1,396.8 minutes.
- **Burn factor: 33.3x** the full monthly budget in a single event — the same result Module 1,
  lesson 6 first calculated with a hypothetical SLO, now confirmed against Andes Cargo's actual
  chosen SLO.
- Burn rate (the *rate* of consumption, not the total budget consumed) is not calculable for this
  incident — no measurable traffic exists once the infrastructure that would generate it is gone
  (Module 6, lesson 7, Step 4).

## What this document does not do

It does not analyze root cause — that is Module 7's postmortem, built on top of this exact
document. It does not assign blame to any individual — the roles section names functions
performed in this exercise, not a judgment on how DataTalks.Club, a one-person team, actually
responded in real life (Module 6, lesson 5, Step 2, is explicit about that distinction). It does
not claim the mitigation path (an AWS-side snapshot not listed in console) is a reproducible
safety net for any future incident — Module 6, lesson 6's closing point, and Module 7's own
honest backup/restore attempt against LocalStack (lesson 7 of that module), exist precisely
because this mechanism was never guaranteed.

## Consequences

Module 7's postmortem cites this document directly for its summary, impact, and timeline sections
— it does not reconstruct any of these facts from scratch. Module 8's capstone runs a new,
synthetic incident through the same five-stage machine this document already proved works on a
real case.

Step 3 — Verifying the document

wc -l TIMELINE.md
grep -c '^## ' TIMELINE.md
grep -c 'T+' TIMELINE.md

What to expect (literal — you assembled the content, the shape is deterministic):

102
8
16

One hundred two lines, eight sections (A note on timestamps, Timeline, Severity, Roles, Mitigation decision tree, Error budget, What this document does not do, Consequences), and sixteen relative timestamp references — not a single real clock time in the entire document, the same deterministic verification pattern SLO.md and INCIDENT-RESPONSE-PLAN.md already established.

How to read this document, six months later

The same test SLO.md and INCIDENT-RESPONSE-PLAN.md already passed: facing this document, without asking anyone or rereading any lesson from this module, a new reader should be able to answer five questions. What happened, and in what order? (Timeline section, fourteen facts, each with its relative mark). How serious was it, with a verifiable criterion, not an opinion? (Severity, with SEV1's independent criterion's four conditions confirmed). Who responded, with what role, and with what communication cadence? (Roles). How was the data recovered, step by step, with every real branch? (Mitigation decision tree). And how much did it cost, in this guide's formal vocabulary? (Error budget, 33.3x, with the honest caveat that burn rate doesn't apply here). If any of those five questions requires rereading a whole lesson from this module, the document failed its purpose.

Module 6's close

With TIMELINE.md finished, this module delivers exactly what it promised in lesson 1: not a fifth narration of the Claude Code incident, but INCIDENT-RESPONSE-PLAN.md's (Module 5) complete framework operated, step by step, against that real incident — instant detection, declaration with real roles assigned, a response that exhausted its own options within minutes, a mitigation tree with a single genuinely uncertain branch, and a resolution verified at T+24h00m — measured with the complete formal SLI/SLO/error budget vocabulary this guide has built since Module 1. The result — 33.3x the monthly budget, the same figure since the first time it was calculated — is no longer just an isolated number from Module 1: it's the consistent result of a real calculator, applied twice, over the same SLO Andes Cargo chose with evidence.

You enter Module 7 with everything you need already built: TIMELINE.md is the direct input for the blameless postmortem that module writes — the "why it happened" question this module deliberately left unanswered.

Common mistakes

Treating this document as the place where root cause gets investigated, instead of as the record of facts and response Module 7 is going to analyze (repeating the error already named in lesson 4, now applied to the complete document). What happens: someone, finishing this document, starts writing conclusions about "why" the human didn't stop the agent, or "what should change" in the process. How to spot it: if your version of TIMELINE.md includes any root-cause or follow-up-action section. How to fix it: the "What this document does not do" section is explicit — root-cause analysis and action items belong to Module 7's postmortem, built on top of this document, not inside it. Mixing both documents dilutes each one's specific function: this is the factual record; the postmortem is the systemic learning.

Copying this document without having run lesson 7's script yourself, trusting that the "Error budget" section's figures are correct (repeating the same error already named with SLO.md and INCIDENT-RESPONSE-PLAN.md). What happens: someone pastes this document's "Error budget" section without having verified, by running their own copy of the script, that 33.3x and -1,396.8 minutes are the real result. How to spot it: if you can't reproduce, by running your own code, exactly the same numbers cited here. How to fix it: every figure in the "Error budget" section comes from a script you already wrote and ran in this module's lesson 7 — before accepting this document as final, confirm your own run produces, digit for digit, the same results.

Assuming the "Roles" section of this document means Ana and Bruno are real people who responded to DataTalks.Club's incident (repeating, for the third time in this module, the same boundary error between real case and exercise). What happens: someone, reading this document in isolation, without the context of previous lessons, concludes the incident had a real Incident Commander and a real Operations Lead during the original events. How to spot it: if your summary of this document doesn't distinguish between what really happened (the "Timeline" section's left column, up to T+24h00m) and what this exercise adds to it (Andes Cargo's role assignment). How to fix it: the document's header declares it in the second line — "Real incident, external: happened to DataTalks.Club, not to Andes Cargo" — precisely so no future reader loses this distinction, even reading only this document with no additional context.

Exercises

Exercise 1 — Verify this document's "Error budget" section against your own run of lesson 7's script. Confirm you get exactly 33.3x as the burn factor and -1396.8 minutes as the remaining budget.

See solution

Running lesson 7's script (error_budget_minutes(0.999, 43200) followed by the operations over INCIDENT_MINUTES = 1440) gives: budget_minutes = 43.2, minutes_consumed = 1440, minutes_remaining = 43.2 - 1440 = -1396.8, burn_factor = 1440 / 43.2 ≈ 33.3. It matches this document's "Error budget" section exactly. This exercise confirms that the portfolio document you just assembled is traceable to an executable script, not an unsupported claim — anyone auditing this document can run the same code and land in exactly the same place.

Exercise 2 — Defend this document's "Severity" section against a real objection. A technical interviewer asks: "if burn rate can't be calculated for this incident, how do you know for certain it's SEV1 and not, say, SEV2?" How do you respond, using the document itself?

See solution

A complete answer: "This document's Severity section already answers that directly — INCIDENT-RESPONSE-PLAN.md's matrix doesn't depend solely on burn rate; it includes an independent criterion for irrecoverable data loss without external support intervention, designed precisely for cases like this where there's no measurable traffic. We verified that criterion's four conditions, fact by fact, against the timeline: there was real data loss, it wasn't recoverable with the team's own tools, it required escalating to external AWS support, and the system wasn't even responding to anything else — a case more severe than the minimum bar the matrix requires for SEV1. The absence of a burn rate number doesn't leave the classification hanging; the independent criterion is just as rigorous, only verifiable with facts instead of a formula."

Exercise 3 — Explain why the "What this document does not do" section explicitly includes "it does not claim the mitigation path... is a reproducible safety net for any future incident," instead of letting that conclusion be inferred from the "Mitigation decision tree" section.

See solution

Without that explicit statement, someone reading only the mitigation decision tree — which ends in a successful recovery — might incorrectly infer that the mechanism that worked this time (an internal AWS snapshot not listed in the console) is a reliable guarantee for the future. Module 6, lesson 6 already precisely established that this mechanism was never guaranteed by any explicit configuration on the customer's side — it was, to a large extent, an unplanned favorable outcome. Explicitly declaring that limitation, instead of leaving it implicit, prevents this document from being read as a recommendation to "trust that AWS will have a backup snapshot" — the same honesty discipline every document in this guide applies to its own limits, and the direct reason Module 7 builds, with LocalStack, an explicit backup/restore attempt for Andes Cargo, instead of depending on the same luck.

Summary and next step

In this module's final project you assembled TIMELINE.md's complete version: fourteen chronological facts with relative timestamps (lesson 2), the SEV1 classification with its independent criterion verified (lesson 3), the assigned roles with their communication cadence (lessons 4 and 5), the complete mitigation decision tree (lesson 6), and the error budget recalculated with the real tool — 33.3x, -1,396.8 minutes of deficit (lesson 7). You verified the document with the same deterministic discipline as this entire ecosystem: 102 lines, 8 sections, 16 relative timestamps, not a single real clock time.

Before closing this module you should be able to: recite the complete timeline, from T-~1h to T+24h00m, without looking at the document; justify the SEV1 severity with the independent criterion, not with burn rate; and explain, with the document itself, why "operating" this incident produced five distinct artifacts, none of which repeats the narration three sibling guides already built.

With this, sre-and-incident-response-guide's Module 6 is complete: the Claude Code incident, operated start to finish through Andes Cargo's incident response framework, with the specific angle this guide promised since Module 1 — recovery, measured with SRE's complete formal vocabulary. Module 7 takes TIMELINE.md, exactly as it stands here, and writes on top of it the complete blameless postmortem: the root cause, what went right and what went wrong, and the action items with an owner and priority that no previous module had built yet.

Resources

  1. This module, lessons 2 through 7 — the direct source of every section of this document.
  2. This same repository, Module 5, lesson 8 (08-project-andes-cargos-incident-response-plan.md) — INCIDENT-RESPONSE-PLAN.md, the complete framework this document demonstrates operated.
  3. Alexey Grigorev — How I Dropped Our Production Database — the primary source for every fact verified in this document.
  4. Google SRE — Incident Management Guide — the five-stage lifecycle this document, start to finish, demonstrates applied to a real case.
← Previous7. This incident's error budget, with the full vocabularyNext →1. Introduction: from the timeline to systemic learning