7. Operational Runbook

Description

It's 3am. An alert fired. Your service is degraded. The runbook is the difference between you resolving the problem in 5 minutes or spending 2 hours trying to remember how the system works under pressure. A runbook isn't architecture documentation — it's your immediate action guide that answers "what do I do now?" for each type of incident. In this capsule you'll create the complete runbook for your Production AI System, with ready-to-copy commands and decision trees you can follow without thinking. When you finish, you'll have a document that saves you real time when something goes wrong — because in an incident, your ability to think clearly is reduced, and the runbook thinks for you.

What a good runbook has (and what it doesn't)

❌ Runbook that's useless at 3am:
   "The system uses clean architecture with DI..."
   "The reliability layer is composed of..."
   "The data model is..."

✅ Runbook that's actually useful:
   INCIDENT: High latency (requests taking > 10 seconds)
   
   TYPICAL SYMPTOMS:
   - Alerts of p99 > 15s
   - Users report "infinite spinner"
   - Health /ready returns 200 but requests are slow
   
   DIAGNOSIS (run in this order):
   1. curl http://app/health/deps → is OpenAI responding?
   2. jq 'select(.duration_ms > 10000)' logs/app.json | tail -5 → which endpoints?
   3. Check https://status.openai.com → is there an incident?
   
   RESOLUTION:
   - If it's an OpenAI incident → communicate to users, wait
   - If the circuit is open → the fallback should activate (verify degraded=true in responses)
   - If there's no incident but it's slow → check prompt length, max_tokens config
   
   ROLLBACK: Reduce max_tokens in .env if prompts are too long, redeploy

Complete RUNBOOK.md

# Operational Runbook — Production AI System

Version: 1.0
Last updated: [DATE]

---

## Contact and escalation information

| Level | When to escalate | Contact |
|-------|----------------|----------|
| L1: On-call | First responder | [Slack channel #alerts] |
| L2: Senior eng | Problem persists > 15min | [PagerDuty] |
| L3: Lead | Impact > 50% of users, cost > $200 | [Direct phone] |

**SLO target**: < 1% of requests with an error in 5-minute windows

---

## Quick diagnosis commands

\`\`\`bash
# General system state:
curl -s http://APP_URL/health/deps | python -m json.tool

# Last 20 errors:
tail -200 logs/app.json | jq 'select(.level == "error")' | tail -20

# Latency in the last hour:
tail -1000 logs/app.json | jq 'select(.event == "request_completed") | .duration_ms' | awk '{s+=$1; n++} END {print "avg:", s/n, "ms"}'

# Today's cost:
today=$(date +%Y-%m-%d)
jq --arg d "$today" 'select(.timestamp | startswith($d)) | .cost_usd' logs/app.json | awk '{s+=$1} END {print "Total: $" s}'

# Circuit breaker status:
curl -s http://APP_URL/health/deps | jq '.checks.circuit_breakers'

# Fallback activation in the last hour:
tail -200 logs/app.json | jq 'select(.event == "fallback_provider_used")' | wc -l
\`\`\`

---

## Incident 1: High Latency

**Severity**: Medium-High  
**SLO impact**: p99 > 15s for > 5 minutes

**Symptoms:**
- Alerts of p99 latency > 15,000ms
- Users report "infinite loading"
- health/live returns 200, but requests are slow

**Diagnosis (in order):**

\`\`\`bash
# Step 1: Check if OpenAI is slow
curl -s http://APP_URL/health/deps | jq '.checks.openai'
# → "latency_ms": 8000  # OpenAI is slow
# → "status": "error"   # OpenAI is down

# Step 2: Check if the circuit breaker is tripped
curl -s http://APP_URL/health/deps | jq '.checks.circuit_breakers'
# → If any circuit shows "state": "open", the fallback should activate

# Step 3: See the slowest requests
tail -500 logs/app.json | jq 'select(.event == "request_completed") | {duration_ms, request_id}' | jq -s 'sort_by(.duration_ms) | reverse | .[0:5]'

# Step 4: Check if there are retries (a sign of transient errors)
tail -200 logs/app.json | jq 'select(.event == "llm_retry_attempt")' | wc -l
\`\`\`

**Resolution:**

| Cause | Action |
|-------|--------|
| OpenAI incident | Communicate to users. Verify fallback is active. Wait. |
| Circuit open but no fallback | See incident 3 (5xx errors) |
| Prompts too long | Reduce `max_tokens` in config, redeploy |
| Rate limit | See incident 5 (rate limit) |

**User communication** (if > 5 min):
> "We're experiencing high latency in the analysis service. The team is investigating. ETA for resolution: 15-30 min."

---

## Incident 2: Abnormally High Cost

**Severity**: High (direct financial impact)  
**Typical trigger**: Cost/day alert > $X

**Symptoms:**
- Alert from the budget limiter or from OpenAI's billing
- Many tokens per request in logs
- Possible abuse (many requests from one source)

**Diagnosis:**

\`\`\`bash
# Step 1: See cost per request
tail -100 logs/app.json | jq 'select(.cost_usd != null) | .cost_usd' | sort -n | tail -20

# Step 2: Identify the most expensive requests
tail -500 logs/app.json | jq 'select(.cost_usd > 0.05) | {request_id, cost_usd, input_tokens, output_tokens}' | head -20

# Step 3: Check if there are many requests from one source (abuse)
tail -1000 logs/app.json | jq -r '.client_ip // "unknown"' | sort | uniq -c | sort -rn | head -10

# Step 4: See today's total
today=$(date +%Y-%m-%d)
jq --arg d "$today" 'select(.timestamp | startswith($d)) | .cost_usd // 0' logs/app.json | awk '{s+=$1} END {print "Today: $" s}'
\`\`\`

**Resolution:**

| Cause | Action |
|-------|--------|
| Bug in prompt (generates extra tokens) | Review the last deploy, rollback if necessary |
| Abuse (many requests) | Rate limit per IP, block the source if applicable |
| Very long input (many tokens) | Add input truncation before the LLM |
| max_tokens unnecessarily high | Reduce max_tokens in config |

**If the cost keeps rising and can't be stopped:**
1. Enable `USE_MOCK_PROVIDER=true` temporarily (the service degrades to mock, zero cost)
2. Or disable the affected endpoint until resolved

---

## Incident 3: 5xx Errors (Rate > 1%)

**Severity**: High  
**SLO impact**: error rate > 1% for > 5 minutes

**Symptoms:**
- Users report "Server error"
- Error rate alerts
- health/ready may return 503

**Diagnosis:**

\`\`\`bash
# Step 1: Complete health check
curl -v http://APP_URL/health/ready
curl -v http://APP_URL/health/deps

# Step 2: See recent errors with context
tail -200 logs/app.json | jq 'select(.level == "error") | {event, error_type, error_message: .error_message[:200], request_id}' | head -10

# Step 3: Find a specific failed request to trace
FAILED_REQUEST_ID="abc123"
grep "$FAILED_REQUEST_ID" logs/app.json | jq .

# Step 4: See the circuit breaker state
curl -s http://APP_URL/health/deps | jq '.checks.circuit_breakers'

# Step 5: Check if the fallback is activating (a sign the primary is failing)
tail -100 logs/app.json | jq 'select(.event == "fallback_provider_used")' | wc -l
\`\`\`

**Decision tree:**

\`\`\`
Does health/deps return an error on openai?
├── YES (OpenAI down):
│   ├── Is the circuit breaker OPEN? → the fallback should activate
│   │   ├── YES and the fallback works → requests degrade but don't fail → OK
│   │   └── YES but the fallback also fails → critical incident, escalate
│   └── NO (circuit open but OpenAI says it's fine) → race condition, wait
└── NO (OpenAI says it's fine but there are errors):
    ├── Are the errors parse errors? (event: "json_parse_completely_failed")
    │   → Prompt issue, review the last prompt change
    └── Are the errors auth errors? (event: "llm_call_failed", error_type: AuthenticationError)
        → API key issue, verify OPENAI_API_KEY in env
\`\`\`

---

## Incident 4: Guardrails Blocking Legitimate Inputs (False Positives)

**Severity**: Medium  
**Symptoms:**
- Users report "My message was rejected"
- Many `guardrail_activated` logs for requests that look legitimate

**Diagnosis:**

\`\`\`bash
# See which inputs are being blocked
tail -200 logs/app.json | jq 'select(.event == "input_guardrail_blocked") | {reason, text_preview, request_id}'

# Guardrail activation rate
total=$(tail -1000 logs/app.json | jq 'select(.event == "request_completed")' | wc -l)
blocked=$(tail -1000 logs/app.json | jq 'select(.event == "input_guardrail_blocked")' | wc -l)
echo "Guardrail activation rate: $blocked / $total"
\`\`\`

**Resolution:**

| Cause | Action |
|-------|--------|
| Pattern too aggressive | Adjust `injection_sensitivity` in config: `medium` → `low` |
| New common false positive | Add exceptions in the guardrail |
| Bug in guardrails update | Rollback the deploy |

**Note**: Don't disable guardrails completely. Adjust the threshold or add specific exceptions.

---

## Incident 5: OpenAI Rate Limit (Massive 429s)

**Severity**: Medium  
**Symptoms:**
- Many `llm_retry_attempt` with error_type `RateLimitError` in logs
- Latency increases (due to backoff)
- The circuit may start to open

**Diagnosis:**

\`\`\`bash
# See the frequency of rate limit errors
tail -500 logs/app.json | jq 'select(.event == "llm_retry_attempt" and .exception_type == "RateLimitError")' | wc -l

# See if the client-side rate limiter is helping
tail -200 logs/app.json | jq 'select(.event == "client_rate_limit_rejected")' | wc -l
# If this number is high, the rate limiter is working but insufficient

# See the request rate per minute
tail -1000 logs/app.json | jq '.timestamp' | cut -c1-16 | sort | uniq -c | sort -rn | head -10
\`\`\`

**Resolution:**

| Cause | Action |
|-------|--------|
| Traffic spike | The rate limiter should be throttling. If not: adjust `MAX_REQUESTS_PER_MINUTE` |
| Rate limit too high in settings | Reduce `MAX_REQUESTS_PER_MINUTE` to 60% of OpenAI's real limit |
| Tier change in OpenAI | Verify current limits at platform.openai.com/limits |

---

## Incident 6: Broken Deploy (Post-Deploy Issues)

**Severity**: Critical  
**Symptoms:**
- health/live returns 500 (the process didn't start)
- health/ready returns 503 (startup checks failed)
- Configuration errors in the first logs

**Diagnosis:**

\`\`\`bash
# See the first logs at startup
head -20 logs/app.json | jq .

# Look for startup errors
grep "startup\|RuntimeError\|ValueError" logs/app.json | head -10

# Verify the configuration
python -c "from src.config import get_settings; get_settings()"
\`\`\`

**Immediate resolution:**

\`\`\`bash
# Rollback to the previous deploy (Kubernetes):
kubectl rollout undo deployment/production-ai-system
kubectl rollout status deployment/production-ai-system

# Verify that the rollback worked:
curl http://APP_URL/health/live
python scripts/post_deploy_check.py --no-monitor
\`\`\`

---

## Complete rollback procedure

\`\`\`bash
# 1. Identify the previous version:
kubectl rollout history deployment/production-ai-system

# 2. Do the rollback:
kubectl rollout undo deployment/production-ai-system

# 3. Verify that it's running:
kubectl rollout status deployment/production-ai-system

# 4. Smoke test:
curl http://APP_URL/health/live
curl http://APP_URL/health/ready
curl -X POST http://APP_URL/api/v1/analyze \
  -H "Content-Type: application/json" \
  -d '{"text": "test"}'

# 5. If the smoke test passes, the rollback was successful.
# 6. If it fails: escalate to L3 and/or disable the service temporarily.
\`\`\`

---

## Post-Incident: What to do after resolving an incident?

In the next 24 hours:

1. Write a brief post-mortem (5 min — not a novel):

   • What happened?
   • How long did it last?
   • Why did it happen? (root cause, not symptom)
   • What will prevent it from happening again?

2. Add the new incident type to this runbook if it wasn't there

3. If there was user impact, communicate the resolution

Minimum post-mortem template:

INCIDENT: [date and time]
DURATION: X minutes
IMPACT: X% of users affected
ROOT CAUSE: [one sentence]
ACTIONS:
  - [X] Fixed in the deploy of [date]
  - [ ] TODO: [preventive improvement] — assigned to [name]

---

## Exercises

### Exercise 1: Simulate the diagnosis of Incident 1

In your local system, add a mock that delays the response by 8 seconds and verify that:
1. You see the high-latency logs
2. You know which command to use to find them
3. The circuit breaker eventually opens

<details>
<summary>See guide</summary>

```python
# In mock_provider.py, add a delay:
import time

class SlowMockProvider:
    def complete(self, messages, **kwargs) -> str:
        time.sleep(8)  # Simulate high latency
        return '{"sentiment": "positive", "score": 0.8, "confidence": 0.9}'

# In dependencies.py (temporarily):
if settings.use_slow_mock:
    return SlowMockProvider()

# Command to see the slow requests:
tail -50 logs/app.json | jq 'select(.duration_ms > 5000) | {duration_ms, request_id}'

Exercise 2: Write a post-mortem for a simulated incident

Simulate the following scenario: at 14:30, the error rate rose to 8% for 12 minutes. The cause was that someone changed the API key in .env to an invalid value during a deploy. The fallback activated correctly and served degraded responses.

Write the post-mortem using this capsule's template. Include: timeline, root cause, real impact, and at least 2 preventive actions.

## Post-Mortem: Error Rate Spike — 2026-03-08

**INCIDENT**: 2026-03-08 14:30 UTC
**DURATION**: 12 minutes (14:30 - 14:42)
**IMPACT**: 8% error rate. ~60% of requests served degraded responses (fallback active).
  Users received sentiment="unknown" instead of the real analysis.
**SEVERITY**: Medium (service degraded, not down)

### Timeline
- 14:25 — Deploy to production with a new .env
- 14:30 — Error rate alerts > 3%
- 14:31 — On-call checks health/deps → OpenAI shows "status: error, error: AuthenticationError"
- 14:33 — Circuit breaker opens. Fallback provider active. Error rate drops to 2% (only the
  requests between the deploy and the circuit opening failed)
- 14:35 — On-call identifies that the API key in .env is invalid (it was copied wrong from the vault)
- 14:38 — Rollback of the .env to the previous value
- 14:40 — Redeploy with the correct API key
- 14:42 — health/deps shows OpenAI "status: ok". Circuit breaker closes. Service normal.

### Root cause
The OpenAI API key was copied incorrectly during the secrets rotation.
The value in .env had an extra character at the end (an invisible newline).

### What worked well?
- The circuit breaker opened in ~2 minutes, limiting the blast radius
- The fallback provider served degraded responses — users didn't see 500 errors
- The logs clearly showed "AuthenticationError" with a request_id to correlate

### Preventive actions
- [x] Add an API key validation test in pre_launch_validation.py:
  make a test request to OpenAI with the key before deploying
- [ ] Automate the secrets rotation from the vault (instead of copying manually)
- [ ] Add a check in startup.py that makes a test LLM call and fails
  if the key is invalid (fail-fast instead of discovering it in production)

Exercise 3: Add a new incident type to the runbook

Your system now has a new /api/v1/batch-analyze endpoint that processes up to 50 texts in a single request. Write the runbook entry for the incident: "Batch endpoint consumes too many tokens and drives up costs".

Include: severity, symptoms, diagnosis (with commands), resolution, and user communication.

## Incident 7: Batch Endpoint — Excessive Token Consumption

**Severity**: High (financial impact)
**Typical trigger**: Cost/hour alert > $X, or batch requests with > 10,000 input tokens

**Symptoms:**
- Cost per request of the batch endpoint > $0.50
- Alerts from the budget limiter
- Logs show `input_tokens > 10000` for requests to the batch endpoint

**Diagnosis:**

\```bash
# Step 1: See the most expensive batch requests
tail -500 logs/app.json | jq '
  select(.event == "request_completed" and .endpoint == "/api/v1/batch-analyze")
  | {request_id, cost_usd, input_tokens, output_tokens, items_count}
' | jq -s 'sort_by(.cost_usd) | reverse | .[0:5]'

# Step 2: See the distribution of batch sizes
tail -1000 logs/app.json | jq '
  select(.endpoint == "/api/v1/batch-analyze") | .items_count
' | sort -n | uniq -c | sort -rn

# Step 3: See if a specific client is abusing it
tail -500 logs/app.json | jq '
  select(.endpoint == "/api/v1/batch-analyze") | .client_ip
' | sort | uniq -c | sort -rn | head -5

# Step 4: Total cost of the batch endpoint today
today=$(date +%Y-%m-%d)
jq --arg d "$today" '
  select(.timestamp | startswith($d))
  | select(.endpoint == "/api/v1/batch-analyze")
  | .cost_usd // 0
' logs/app.json | awk '{s+=$1} END {print "Batch cost today: $" s}'
\```

**Resolution:**

| Cause | Action |
|-------|--------|
| Texts too long in the batch | Add per-item truncation (max 500 chars) |
| Batch size with no limit | Add validation: max 20 items per batch |
| Abuse from one client | Rate limit per IP for the batch endpoint |
| Bug that duplicates items | Review the last deploy of the batch handler |

**If the cost keeps rising:**
1. Temporarily disable the batch endpoint (return 503 "maintenance")
2. Keep the individual endpoint active
3. Investigate and fix before reactivating

Exercise 4: Create an automatic diagnosis script

Create scripts/diagnose.py that automatically runs the first 3 diagnosis steps of each incident and generates a report. The script must:

1. Check health endpoints
2. Count errors in the last 100 logs
3. Check the circuit breaker state
4. Show a summary with a recommendation of which incident to investigate

# scripts/diagnose.py
"""
Automatic diagnosis of the Production AI System.
Runs the most common checks and suggests what to investigate.

Usage:
  python scripts/diagnose.py
  python scripts/diagnose.py --url http://staging.example.com
"""
import json
import urllib.request
import urllib.error
import sys
import argparse
from pathlib import Path

def check_health(base_url: str) -> dict:
    """Checks the health endpoints."""
    results = {}
    for endpoint in ["/health/live", "/health/ready", "/health/deps"]:
        try:
            resp = urllib.request.urlopen(f"{base_url}{endpoint}", timeout=10)
            body = json.loads(resp.read())
            results[endpoint] = {"status": resp.status, "body": body}
        except urllib.error.HTTPError as e:
            results[endpoint] = {"status": e.code, "error": str(e)}
        except Exception as e:
            results[endpoint] = {"status": 0, "error": str(e)}
    return results

def analyze_recent_logs(log_file: str = "logs/app.json", n_lines: int = 100) -> dict:
    """Analyzes the most recent logs."""
    log_path = Path(log_file)
    if not log_path.exists():
        return {"error": f"Log file not found: {log_file}"}

    lines = log_path.read_text().strip().split("\n")[-n_lines:]
    errors = 0
    slow_requests = 0
    fallbacks = 0
    rate_limits = 0

    for line in lines:
        try:
            entry = json.loads(line)
            if entry.get("level") == "error":
                errors += 1
            if entry.get("duration_ms", 0) > 5000:
                slow_requests += 1
            if entry.get("event") == "fallback_provider_used":
                fallbacks += 1
            if "RateLimitError" in str(entry.get("exception_type", "")):
                rate_limits += 1
        except json.JSONDecodeError:
            continue

    return {
        "total_analyzed": len(lines),
        "errors": errors,
        "slow_requests": slow_requests,
        "fallbacks": fallbacks,
        "rate_limits": rate_limits,
    }

def suggest_investigation(health: dict, logs: dict) -> list[str]:
    """Based on the data, suggests what to investigate."""
    suggestions = []

    live = health.get("/health/live", {})
    ready = health.get("/health/ready", {})

    if live.get("status") != 200:
        suggestions.append("🔴 CRITICAL: /health/live fails → the process isn't running. See Incident 6 (broken deploy).")
    if ready.get("status") != 200:
        suggestions.append("🟡 ALERT: /health/ready fails → startup checks didn't pass. Verify API key and config.")

    if isinstance(logs, dict) and "error" not in logs:
        if logs["errors"] > 5:
            suggestions.append(f"🟡 {logs['errors']} errors in the last {logs['total_analyzed']} logs → See Incident 3 (5xx errors).")
        if logs["slow_requests"] > 3:
            suggestions.append(f"🟡 {logs['slow_requests']} slow requests (>5s) → See Incident 1 (high latency).")
        if logs["fallbacks"] > 2:
            suggestions.append(f"🟡 {logs['fallbacks']} fallback activations → The primary provider has problems.")
        if logs["rate_limits"] > 0:
            suggestions.append(f"🟡 {logs['rate_limits']} rate limit errors → See Incident 5 (OpenAI rate limit).")

    if not suggestions:
        suggestions.append("✅ No obvious problems detected. The system seems healthy.")

    return suggestions

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Automatic diagnosis")
    parser.add_argument("--url", default="http://localhost:8000")
    parser.add_argument("--logs", default="logs/app.json")
    args = parser.parse_args()

    print("\n🔍 Running automatic diagnosis...\n")

    print("1. Health checks:")
    health = check_health(args.url)
    for endpoint, result in health.items():
        status = result.get("status", "?")
        icon = "✅" if status == 200 else "❌"
        print(f"   {icon} {endpoint} → {status}")

    print("\n2. Recent logs analysis:")
    logs = analyze_recent_logs(args.logs)
    if "error" in logs:
        print(f"   ⚠️  {logs['error']}")
    else:
        print(f"   Logs analyzed: {logs['total_analyzed']}")
        print(f"   Errors: {logs['errors']}")
        print(f"   Slow requests: {logs['slow_requests']}")
        print(f"   Fallbacks: {logs['fallbacks']}")
        print(f"   Rate limits: {logs['rate_limits']}")

    print("\n3. Recommendations:")
    suggestions = suggest_investigation(health, logs)
    for s in suggestions:
        print(f"   {s}")
    print()

Troubleshooting

Problem: The runbook's jq commands fail with "parse error"

Symptoms:

• jq: error (at <stdin>:1): Expected value
• The runbook commands return nothing or give a parsing error

Most likely cause: Your log file isn't valid JSON Lines. Each line must be an independent JSON object. If structlog isn't configured for JSON output, the logs may be plain text.

Solution:

# Verify that the logs are valid JSON:
head -1 logs/app.json | python -m json.tool

# If it fails, review the structlog configuration:
# In logging_config.py, make sure you use JSONRenderer:
# structlog.configure(
#     processors=[..., structlog.processors.JSONRenderer()]
# )

# If the logs are plain text, use grep instead of jq:
grep "error" logs/app.json | tail -10

Problem: The circuit breaker never opens despite continuous errors

Symptoms:

• The logs show many llm_call_failed in a row
• But the circuit breaker stays in the "closed" state
• The fallback never activates

Most likely cause: The circuit breaker is being instantiated anew on each request (not a singleton). If a new one is created each time, it never accumulates enough failures to open.

Solution:

# In dependencies.py, the circuit breaker must live OUTSIDE the function:

# ❌ Incorrect: new circuit breaker on every request
def build_llm_provider():
    cb = CircuitBreaker(failure_threshold=5, reset_timeout=60)  # New each time
    return CircuitBreakerProvider(OpenAIProvider(...), cb)

# ✅ Correct: singleton that persists across requests
_circuit_breaker = CircuitBreaker(failure_threshold=5, reset_timeout=60)

def build_llm_provider():
    return CircuitBreakerProvider(OpenAIProvider(...), _circuit_breaker)

Problem: The runbook commands use paths that don't exist in your system

Symptoms:

• tail logs/app.json → "No such file or directory"
• curl http://APP_URL/health/deps → "Connection refused"

Most likely cause: The paths and URLs in the runbook template are placeholders that you need to adapt to your environment.

Solution: Before using the runbook in a real incident, personalize it:

# 1. Replace APP_URL with your real URL:
sed -i 's|APP_URL|localhost:8000|g' docs/RUNBOOK.md  # development
sed -i 's|APP_URL|api.myapp.com|g' docs/RUNBOOK.md    # production

# 2. Verify that the logs path exists:
ls -la logs/app.json
# If your logs are elsewhere, update the runbook

# 3. Verify that the health endpoints exist:
curl http://localhost:8000/health/live
curl http://localhost:8000/health/ready

The runbook should be tested in calm, not during an incident. Run each command at least once while everything works to verify that the paths and URLs are correct.

Problem: The post-mortem template doesn't feel useful for your team

Symptoms:

• The template is too formal or too informal
• No one writes them after incidents
• The post-mortems don't generate concrete actions

Most likely cause: The template doesn't fit your team's culture. A long, formal template demotivates; one that's too short doesn't capture what's important.

Solution: Adapt the template to your team's size:

# For teams of 1-3 people (only the essentials):
**What happened**: [1 sentence]
**How long it lasted**: [X min]
**Why**: [root cause]
**What we're going to do**: [1-2 actions with a date]

# For larger teams (add):
**Detailed timeline**: minute by minute
**Impact on metrics**: error rate, affected users
**Communication**: what was communicated and when
**Runbook review**: did the runbook cover this case?

The most important rule: the post-mortem is written in the first 24 hours. After that, the details are forgotten and it loses value.

Summary

• The runbook is for action, not for understanding: in an incident, the goal is to resolve, not to learn
• Order matters: the diagnosis steps go from fastest to slowest, from general to specific
• Ready-to-copy commands: it shouldn't require thinking about which command to use
• Real scenarios: cover the 6 most common incidents for AI apps
• Fast post-mortem: document to avoid repeating

Additional resources

1. Google SRE Book — Incident Management — The industry standard
2. PagerDuty Incident Response Guide — Practical guide
3. Postmortem Template (Google) — How to make useful post-mortems
4. jq Manual — For the log query commands