5. Operational Runbook — What to Do When Something Fails

Description

In this capsule you'll create an operational runbook: a document with step-by-step procedures to respond to common production incidents. The service goes down, the deploy fails, costs spike, latency degrades — for each scenario, you'll have a clear procedure with diagnosis, resolution, and prevention. A system without a runbook is a system only you can operate — and that doesn't scale.

Context: So far you deployed and validated. But deployment doesn't end when the app is online — it ends when it's documented and operable. The runbook is the artifact that turns your system from "works while you're around" to "any engineer can operate it." It's the difference between a prototype and a production system.

Why You Need a Runbook

Without a runbook vs with a runbook

WITHOUT A RUNBOOK — Service down at 3am:
1. You get an alert (if you configured alerts)
2. What do I do? Where do I see the logs?
3. Open the laptop, try to remember the commands
4. Google "how to see logs on Railway"
5. 20 min later: find the error
6. 30 min later: apply a fix
7. 45 min of total downtime
8. If you're on vacation: nobody knows what to do

WITH A RUNBOOK — Service down at 3am:
1. You get an alert
2. Open the runbook → Section: "Service not responding"
3. Step 1: Check status → railway status
4. Step 2: View logs → railway logs --lines 100
5. Step 3: If OOM → restart with more memory
6. Step 4: If API key → verify secrets
7. 10 min of total downtime
8. Anyone on the team can follow the steps

The structure of a runbook

docs/runbook.md
├── Metadata (authors, last update, covered services)
├── Access and tools (URLs, credentials, dashboards)
├── Procedures by scenario:
│   ├── Service not responding
│   ├── Deploy fails
│   ├── Degraded latency
│   ├── Unexpected costs
│   ├── Expired/invalid API key
│   └── Emergency rollback
├── Escalation (who to contact if you can't resolve it)
└── Post-mortem template

Complete Runbook Template

Metadata and access

# Operational Runbook — [AI System Name]

**Last updated:** [Date]
**Author:** [Your name]
**Version:** 1.0

## System Information

| Item | Value |
|------|-------|
| Production URL | https://your-app.railway.app |
| Health check | https://your-app.railway.app/health |
| Readiness | https://your-app.railway.app/health/ready |
| Platform dashboard | https://railway.app/project/xxx |
| GitHub repo | https://github.com/your-user/your-repo |
| CI/CD pipeline | https://github.com/your-user/your-repo/actions |
| Monitoring | https://uptimerobot.com/dashboard |

## Credentials and Access

| Service | How to access |
|----------|-------------|
| Railway dashboard | Login with GitHub at railway.app |
| GitHub Actions | Repo → Actions tab |
| OpenAI dashboard | platform.openai.com (team account) |
| UptimeRobot | uptimerobot.com (team email) |

⚠️ API keys and secrets are on the deployment platform.
They are NEVER stored in the repo or in this document.

Procedure 1: Service Not Responding

Symptoms

• UptimeRobot reports downtime
• curl https://your-app.railway.app/health returns an error or timeout
• Users report that the app doesn't work

Diagnosis

# Step 1: Confirm that the service is down
curl -v https://your-app.railway.app/health
# If timeout → the container isn't running
# If 502/503 → the platform is up but the app isn't

# Step 2: Check the status on the platform
# Railway
railway status
railway logs --lines 50

# Render
# Dashboard → your-service → Events → view the last deploy/crash

# Fly.io
flyctl status
flyctl logs --lines 50

# Step 3: Look for the error in the logs
# Common errors:
# - "OOM killed" → ran out of memory
# - "ModuleNotFoundError" → missing dependency
# - "OPENAI_API_KEY" → variable not configured
# - "Address already in use" → port conflict

Resolution

# If OOM (Out of Memory):
# Option A: Restart the service
railway up --detach  # Railway
flyctl restart       # Fly.io

# Option B: Increase memory (if the plan allows it)
# Railway: Settings → Resources → increase memory
# Fly.io: flyctl scale memory 512  # MB

# If missing dependency:
# Verify that requirements.txt is up to date
pip freeze > requirements.txt
git add requirements.txt && git commit -m "fix: update dependencies" && git push

# If missing environment variable:
# Railway
railway variables set OPENAI_API_KEY=sk-...

# Fly.io
flyctl secrets set OPENAI_API_KEY=sk-...

# If the container doesn't start:
# Check detailed logs
railway logs --lines 200 | head -50  # First lines = startup errors

Prevention

• Configure memory alerts on the platform
• Health checks every 5 minutes with UptimeRobot
• Post-deploy smoke tests that verify inference

Procedure 2: Deploy Fails

Symptoms

• The GitHub Actions pipeline shows ❌ on the deploy job
• The platform shows "Deploy failed" in the dashboard
• The service keeps running with the previous version

Diagnosis

# Step 1: See which job failed in GitHub Actions
# GitHub → Actions → latest workflow run → view the logs of the failed job

# Step 2: If it failed at "test"
# The tests don't pass. The deploy didn't run (correct).
# Look at the pytest output to identify the failing test.

# Step 3: If it failed at "build"
# The Docker image doesn't build.
# Common errors:
# - pip install fails → dependency with an incompatible version
# - COPY fails → referenced file doesn't exist
# - Dockerfile syntax error

# Step 4: If it failed at "deploy"
# The deploy trigger didn't work.
# - Expired token/secret → regenerate on the platform
# - Deploy hook URL changed → update in GitHub Secrets
# - The platform has an incident → check the status page

# Step 5: If it failed at "validate"
# The deploy happened but the system doesn't work.
# THIS IS A PROBLEM — you need to decide whether to roll back.

Resolution

# If the test fails:
# 1. Review the error in the GitHub Actions log
# 2. Reproduce locally: pytest tests/ -v
# 3. Fix the test or the code
# 4. Push the fix → the pipeline re-runs

# If the build fails:
# 1. Reproduce locally: docker build -t test .
# 2. Fix the Dockerfile or requirements.txt
# 3. Push the fix

# If the deploy trigger fails:
# 1. Verify that the secret/token is configured
# 2. Verify that it hasn't expired
# Railway: railway login → railway whoami
# Fly.io: flyctl auth token
# 3. Regenerate if it expired and update it in GitHub Secrets

# If validation fails (deploy happened but doesn't work):
# IMMEDIATE ROLLBACK:
# Option A: Revert the commit
git revert HEAD
git push

# Option B: Manual deploy of the previous version
# Railway
railway rollback

# Option C: GitHub Actions → Rollback workflow → Run

Prevention

• Run pytest and docker build locally before pushing
• Renew platform tokens before they expire
• Keep a rollback workflow ready for emergencies

Procedure 3: Degraded Latency

Symptoms

• Smoke tests report latency > target (e.g.: > 5s)
• Users report that the app "is slow"
• Latency metrics show a progressive increase

Diagnosis

# Step 1: Measure current latency
time curl -s -X POST https://your-app.railway.app/api/inference \
    -H "Content-Type: application/json" \
    -d '{"prompt": "Say hello"}'

# Step 2: Identify the cause
# A. Is it the platform?
time curl -s https://your-app.railway.app/health
# If /health takes > 500ms → platform or network problem

# B. Is it the OpenAI API?
time curl -s https://api.openai.com/v1/models \
    -H "Authorization: Bearer $OPENAI_API_KEY"
# If it takes > 2s → OpenAI is slow, not your fault

# C. Is it your code?
# Check if there's memory pressure
# Railway: dashboard → Metrics → Memory usage
# If it's near the limit → the app is swapping

Resolution

# If it's cold start (free tier):
# Platforms with a free tier put the app to sleep after inactivity.
# First request after sleeping = 5-30s of cold start.
# Solution: Upgrade to a paid plan, or accept the cold start.

# If it's memory pressure:
# Option A: Optimize memory usage
# - Reduce batch sizes
# - Use streaming instead of loading the whole response into RAM
# - Limit the context window size

# Option B: Scale vertically
# Railway: Settings → Resources → more RAM
# Fly.io: flyctl scale memory 1024

# If it's the OpenAI API:
# There's not much to do. Options:
# - Implement caching (Redis) for frequent responses
# - Reduce tokens (shorter prompts)
# - Switch to a faster model (gpt-4o-mini vs gpt-4o)

# If it's a bug in your code:
# Basic profiling:
import time

@app.post("/api/inference")
async def inference(request: InferenceRequest):
    t0 = time.time()
    # ... preprocessing
    t1 = time.time()
    # ... LLM call
    t2 = time.time()
    # ... postprocessing
    t3 = time.time()

    timings = {
        "preprocessing_ms": (t1-t0)*1000,
        "llm_call_ms": (t2-t1)*1000,
        "postprocessing_ms": (t3-t2)*1000,
        "total_ms": (t3-t0)*1000,
    }
    # Log timings to identify the bottleneck

Prevention

• Establish performance baselines (capsule 07)
• Monitor latency with periodic smoke tests
• Implement caching for repetitive requests

Procedure 4: Unexpected Costs

Symptoms

• Platform bill higher than expected
• OpenAI bill higher than expected
• Spending alerts (if configured)

Diagnosis

# Step 1: Identify which service generates the cost
# Platform: dashboard → Billing → Usage breakdown
# OpenAI: platform.openai.com → Usage → view by day

# Step 2: Analyze the pattern
# Did traffic increase legitimately?
# → More users = more costs (expected)

# Is there unexpected traffic?
# → Possible bot, scraping, or DDoS

# Does one prompt generate a lot of tokens?
# → A bug that sends very long contexts

# Step 3: Check traffic in the logs
# Railway
railway logs --lines 500 | grep "POST /api/inference" | wc -l
# How many requests in the last few hours?

Resolution

# If it's excessive traffic → implement rate limiting
from slowapi import Limiter
from slowapi.util import get_remote_address

limiter = Limiter(key_func=get_remote_address)

@app.post("/api/inference")
@limiter.limit("10/minute")  # 10 requests per minute per IP
async def inference(request: Request, body: InferenceRequest):
    ...

# If it's a prompt bug → verify the token count
from tiktoken import encoding_for_model

enc = encoding_for_model("gpt-4o-mini")

@app.post("/api/inference")
async def inference(body: InferenceRequest):
    tokens = len(enc.encode(body.prompt))
    if tokens > 2000:
        raise HTTPException(400, "Prompt too long")
    ...

# If you need to stop the service in an emergency
# Railway
railway down

# Fly.io
flyctl scale count 0

# Render: Dashboard → Settings → Suspend service

Prevention

• Configure spending alerts on OpenAI (Settings → Limits)
• Implement rate limiting from day 1
• Establish a cost baseline and review it weekly

Procedure 5: Emergency Rollback

When to activate

ACTIVATE ROLLBACK IF:
├── Health check fails post-deploy and doesn't recover in 5 min
├── Error rate > 50% in production
├── Inference smoke tests fail consistently
└── The system produces incorrect or dangerous responses

DO NOT ACTIVATE ROLLBACK IF:
├── Only a minor endpoint fails, the rest works
├── Latency went up but the service responds
├── It's a cosmetic issue (different response format)
└── The error was present before the deploy (not a regression)

Step-by-step procedure

# === STEP 1: CONFIRM THAT YOU NEED A ROLLBACK ===
# Verify health
curl -s https://your-app.railway.app/health
# Verify inference
curl -s -X POST https://your-app.railway.app/api/inference \
    -H "Content-Type: application/json" \
    -d '{"prompt": "Test"}'

# === STEP 2: IDENTIFY THE LAST GOOD VERSION ===
git log --oneline -10
# Identify the commit before the deploy that broke it

# === STEP 3: RUN THE ROLLBACK ===

# Option A: GitHub Actions (if you have a rollback workflow)
# GitHub → Actions → Rollback → Run workflow
# Input: SHA of the good commit

# Option B: Manual from the platform
# Railway
railway rollback

# Fly.io
flyctl releases
# Identify the good release
flyctl deploy --image registry.fly.io/your-app:release-N

# Render
# Dashboard → Manual Deploy → select the previous commit

# Option C: Git revert
git revert HEAD  # Reverts the last commit
git push main    # Triggers a new deploy

# === STEP 4: VERIFY THE ROLLBACK ===
sleep 60
curl -s https://your-app.railway.app/health
python scripts/smoke_test.py https://your-app.railway.app

# === STEP 5: COMMUNICATE ===
# Notify the team:
# "Rollback executed to [commit]. Cause: [description].
#  Service restored at [time]. Investigating root cause."

# === STEP 6: POST-MORTEM ===
# Create an issue on GitHub with the post-mortem template (see below)

Post-Mortem Template

# Post-Mortem: [Incident title]

**Date:** [Incident date]
**Duration:** [Downtime]
**Severity:** [High/Medium/Low]
**Author:** [Who writes the post-mortem]

## Timeline

| Time | Event |
|------|--------|
| HH:MM | Deploy executed (commit abc123) |
| HH:MM | UptimeRobot alert: service down |
| HH:MM | Engineer investigates, identifies the cause |
| HH:MM | Rollback executed |
| HH:MM | Service restored, smoke tests pass |

## Root Cause

[Technical description of what caused the incident]

## Impact

- Affected users: [number or description]
- Downtime duration: [minutes]
- Failed requests: [estimate]

## What Went Well

- [Monitoring detected the problem in X minutes]
- [The runbook had the correct procedure]
- [The rollback worked]

## What Went Wrong

- [It wasn't detected in the tests]
- [The deploy was done without smoke tests]
- [We didn't have a runbook for this scenario]

## Action Items

- [ ] [Action 1 — who — deadline]
- [ ] [Action 2 — who — deadline]
- [ ] [Action 3 — who — deadline]

Troubleshooting

Problem 1: "I don't have access to the platform when there's an incident"

Cause: Only one person has the credentials.

Solution: Document in the runbook how to access it. Use team accounts wherever possible. On Railway/Render/Fly.io, invite collaborators to the project. Never depend on a single person.

Problem 2: "The runbook has procedures but I don't know which one to apply"

Cause: The symptoms overlap between procedures.

Solution: Add a decision diagram at the start of the runbook:

Does the service respond to /health?
├── NO → Procedure 1: Service not responding
└── YES → Does inference work?
    ├── NO → Procedure 2 or 5 (check if it was post-deploy)
    └── YES → Is the latency acceptable?
        ├── NO → Procedure 3: Degraded latency
        └── YES → Are the costs normal?
            ├── NO → Procedure 4: Unexpected costs
            └── YES → The system is healthy ✅

Problem 3: "After the rollback, I need to deploy the fix but I don't know if it's safe"

Cause: Fear that the fix will also break something.

Solution:

# 1. Verify the fix locally COMPLETELY
docker build -t test . && docker run -d --name test -p 8000:8000 \
    -e OPENAI_API_KEY=$OPENAI_API_KEY test
python scripts/smoke_test.py http://localhost:8000
docker stop test && docker rm test

# 2. If you have staging, deploy there first
# 3. If you don't have staging, merge the fix with confidence
# 4. Monitor the deploy with more attention than usual

Hands-On Exercises

Exercise 1: Create your base runbook

Create the docs/runbook.md file with the metadata, system information, and at least 3 procedures.

# Operational Runbook — My AI System

**Last updated:** 2026-03-08
**Author:** [Your name]
**Version:** 1.0

## System Information

| Item | Value |
|------|-------|
| Production URL | https://my-app.railway.app |
| Health check | https://my-app.railway.app/health |
| Dashboard | https://railway.app/project/xxx |
| Repo | https://github.com/user/repo |
| CI/CD | https://github.com/user/repo/actions |
| Monitoring | https://uptimerobot.com |

## Procedures

### 1. Service not responding
[Copy procedure 1 adapted to your platform]

### 2. Deploy fails
[Copy procedure 2 adapted]

### 3. Emergency rollback
[Copy procedure 5 adapted]

Exercise 2: Simulate an incident

Break something intentionally (change an env var to an invalid value), detect the problem with your health check, and follow your runbook to resolve it.

# 1. Break: Change the API key to an invalid value
# Railway: railway variables set OPENAI_API_KEY=invalid-key

# 2. Wait for the service to update (~30s)
sleep 30

# 3. Detect: Health check
curl -s https://my-app.railway.app/health/ready
# Should show openai: "error"

# 4. Diagnose: Follow Procedure 1 of the runbook
# "Check logs" → railway logs --lines 50
# Should show an OpenAI authentication error

# 5. Resolve: Restore the API key
# railway variables set OPENAI_API_KEY=sk-the-correct-key

# 6. Verify: Smoke tests
sleep 30
python scripts/smoke_test.py https://my-app.railway.app
# Should pass all tests

Exercise 3: Create the decision diagram

Draw the decision diagram that connects symptoms with procedures (like the one shown in troubleshooting).

## Incident Decision Diagram

Does the service respond to /health?
│
├── NO → Was there a recent deploy (<30 min)?
│   ├── YES → Immediate rollback (Proc. 5)
│   └── NO → Check the platform (Proc. 1)
│
└── YES → Does /health/ready show "ready"?
    │
    ├── NO → Which dependency fails?
    │   ├── OpenAI → Verify the API key (Proc. 1 - secrets)
    │   └── Other → Check the specific service
    │
    └── YES → Does inference respond correctly?
        │
        ├── NO → Was it post-deploy?
        │   ├── YES → Rollback (Proc. 5)
        │   └── NO → Debug the code (Proc. 3)
        │
        └── YES → Acceptable latency?
            ├── NO → Proc. 3: Latency
            └── YES → Healthy system ✅

Exercise 4: Write a practice post-mortem

Using the post-mortem template, document the simulated incident from Exercise 2.

# Post-Mortem: Invalid API Key in Production

**Date:** 2026-03-08
**Duration:** ~5 minutes
**Severity:** High (inference wasn't working)
**Author:** [Your name]

## Timeline

| Time | Event |
|------|--------|
| 14:00 | API key changed to an invalid value (simulated) |
| 14:01 | Health/ready check shows OpenAI as "error" |
| 14:02 | Inference smoke test fails |
| 14:03 | Cause identified: invalid API key in logs |
| 14:04 | API key restored |
| 14:05 | Smoke tests pass, service restored |

## Root Cause
The OpenAI API key was changed to an invalid value.

## Impact
- Inference unavailable for ~5 minutes
- Basic health check kept responding 200 (misleading)

## Action Items
- [ ] Improve the health check to fail if OpenAI doesn't connect
- [ ] Add a specific alert for authentication errors
- [ ] Document the API key rotation procedure

Summary

• A runbook turns your system from "only you can operate it" to "anyone can operate it"
• It includes: system metadata, accesses, and step-by-step procedures for common incidents
• The 5 essential procedures: service down, failed deploy, degraded latency, unexpected costs, rollback
• Each procedure has: symptoms, diagnosis, resolution, and prevention
• The decision diagram connects symptoms with procedures — it's the first thing you consult during an incident
• Post-mortems document incidents to learn and prevent recurrences
• The runbook is a living document — update it after every real incident

Additional Resources

1. PagerDuty Incident Response Guide — Complete incident response guide
2. Google SRE Book — Postmortem Culture — The bible of post-mortems
3. Atlassian Incident Management — Incident management handbook
4. Render Incident Response — Troubleshooting on Render
5. Railway Docs — Observability — Logs and observability on Railway
6. Fly.io — Monitoring — Metrics on Fly.io