Module 3: Observability As Sli Input
8. Project: Andes Cargo's observability-to-SLI pipeline
Description
Six lessons built a real pipeline; this project documents it and saves it as two portfolio artifacts. The first, observability/OBSERVABILITY-RUNBOOK.md, answers the question anyone new to the team would ask on day one: "how do we really measure process-shipment-manifest's SLI?" — what gets instrumented, where each piece of data lives, what exact query reads it. The second, observability/dashboard.json, is lesson 6's real Grafana panel, exported as the same file you already used to create it — reproducible, not a screenshot.
Connection to the module
This is the deliverable that closes Module 3 the same way SLO.md closed Module 2: not a prose summary of what you already saw, but the document the rest of the guide is going to cite without re-explaining it. Module 4 reads this same runbook when deciding exactly which metric to build the burn rate alert on. Module 7 cites it in the operational incident runbook.
Step 1 — Why an observability runbook, and not just the code already written
Lessons 3 through 6 already left the complete code: upload_manifest_batch.py, manifest-log-events.json, instrument_manifest_flow.py, manifest_metrics_exporter.py, docker-compose.yml. But code with no map is hard to operate under pressure — exactly the problem a runbook, as Module 7's lesson 5 is going to formalize later, exists to solve. This project doesn't wait until Module 7 to practice that discipline: it documents, in a single short place, the answer to three questions someone would ask in the middle of a real incident — what's being measured? where does that data live right now? what exact command brings it back?
Step 2 — The complete runbook
In andes-cargo-infra/, create observability/OBSERVABILITY-RUNBOOK.md:
# OBSERVABILITY-RUNBOOK.md — How We Measure the process-shipment-manifest SLI
**Status:** Accepted · **Governs:** Module 3 through Module 8 of `sre-and-incident-response-guide`
**Source:** Module 3, lessons 2-7 (the three pillars, real metrics, real logs, real traces,
Prometheus/Grafana, and the calculator run with real telemetry)
**Related:** `SLO.md` (Module 2) defines the SLI this runbook feeds; Module 4 alerts on it.
## What is instrumented
Three pillars, each covering exactly one question a Site Reliability Engineer needs to
answer about `process-shipment-manifest`'s SLI (good events / valid events):
| Pillar | What is instrumented | Question it answers |
|---|---|---|
| Metrics | `AWS/Lambda`/`Invocations`, `AWS/Lambda`/`Errors` (CloudWatch); `manifest_invocations_total`, `manifest_errors_total` (Prometheus) | How many valid events, how many good events? |
| Logs | `/aws/lambda/process-shipment-manifest` — the `Invalid manifest ...` line `lambda_handler` prints before raising, and the `REPORT` line the Lambda runtime appends automatically | Which specific invocation failed, and why? |
| Traces | `observability/instrument_manifest_flow.py` — OTel spans for `shipment-manifest-upload -> process-shipment-manifest -> dynamodb-put-item` | At which exact step of the flow did one specific invocation break? |
## Where the data lives
| Source | Status | Reason |
|---|---|---|
| CloudWatch Metrics (`AWS/Lambda`) | Representative in this repo's authoring environment | No `LOCALSTACK_AUTH_TOKEN` exported here; confirmed on LocalStack's Hobby plan |
| CloudWatch Logs (`/aws/lambda/process-shipment-manifest`) | Representative, same reason | Same as above |
| Prometheus (`manifest_invocations_total`, `manifest_errors_total`) | Real, running | `manifest_metrics_exporter.py` + `docker compose` (`observability/docker-compose.yml`) |
| Jaeger v2 (`jaegertracing/jaeger:2.20.0`) | Real, running | `observability/instrument_manifest_flow.py` + `docker compose` |
| Grafana (`13.1.3`) | Real, running | Dashboard `andes-cargo-manifest-sli`, saved as `observability/dashboard.json` |
## What query reads it
```bash
# Metrics (representative)
awslocal cloudwatch get-metric-statistics --namespace AWS/Lambda --metric-name Errors \
--dimensions Name=FunctionName,Value=process-shipment-manifest \
--start-time 2026-08-14T14:00:00Z --end-time 2026-08-14T15:00:00Z --period 3600 --statistics Sum
# Metrics (real, PromQL)
curl -s 'http://localhost:9090/api/v1/query?query=manifest_errors_total/manifest_invocations_total'
# Logs (representative awslocal + real jq)
awslocal logs filter-log-events --log-group-name /aws/lambda/process-shipment-manifest \
--filter-pattern '?"Invalid manifest" ?"Status: error"'
jq -r '.events[] | select(.message | contains("Status: error")) | .message | capture("RequestId: (?<requestId>[a-f0-9-]+)") | .requestId' manifest-log-events.json
# Traces (real)
curl -s "http://localhost:16686/api/traces?service=andes-cargo-app-server&limit=10"
```
## Known limitations
- The 20-invocation batch used across this module is a deliberate verification batch (3
malformed on purpose), not a sample of real production traffic. Module 3, lesson 7
showed the correct way to use it: as one additional day inside the Module 2 30-day
dataset, never as a standalone monthly SLI.
- CloudWatch metrics and logs are representative in this authoring environment (no
`LOCALSTACK_AUTH_TOKEN`). Prometheus, Grafana, and Jaeger are real and running.
- This runbook documents the SLI-measurement pipeline. It does not alert on it — that is
Module 4's `scripts/burn_rate_evaluator.py` and the Alertmanager/CloudWatch Alarm rules.Step 3 — Verifying the runbook
wc -l observability/OBSERVABILITY-RUNBOOK.md
grep -c '^## ' observability/OBSERVABILITY-RUNBOOK.mdWhat to expect (literal — you wrote the content, the shape is deterministic):
58
4
Fifty-eight lines, four sections (What is instrumented, Where the data lives, What query reads it, Known limitations) — short on purpose, because a runbook nobody reads under pressure isn't doing its job, the same lesson Module 7 is going to develop in depth.
Step 4 — Saving the Grafana panel as a reproducible artifact
In observability/dashboard.json, save exactly the same JSON body you used in lesson 6, Step 6, to create the dashboard:
{
"dashboard": {
"id": null,
"uid": "andes-cargo-manifest-sli",
"title": "Andes Cargo -- process-shipment-manifest SLI input",
"tags": ["andes-cargo", "sre"],
"timezone": "browser",
"panels": [
{
"id": 1,
"type": "stat",
"title": "Invocations vs errors (fixed batch)",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
"targets": [
{ "expr": "manifest_invocations_total - manifest_errors_total", "legendFormat": "good", "refId": "A" },
{ "expr": "manifest_errors_total", "legendFormat": "errors", "refId": "B" }
],
"fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] }
},
{
"id": 2,
"type": "gauge",
"title": "Observed error rate",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 },
"targets": [
{ "expr": "manifest_errors_total / manifest_invocations_total", "legendFormat": "error rate", "refId": "A" }
],
"fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1 }, "overrides": [] }
}
],
"schemaVersion": 39,
"version": 1,
"refresh": ""
},
"overwrite": true
}Verifying the file reproduces the real dashboard exactly (the same command from lesson 6, pointed at the file instead of an inline block):
curl -s -u admin:andescargo -X POST http://localhost:3000/api/dashboards/db \
-H "Content-Type: application/json" \
-d @observability/dashboard.jsonWhat to expect (literal — verified in this environment, the exact same uid and slug from lesson 6):
{"folderUid": "", "id": 3189596175294464, "slug": "andes-cargo-process-shipment-manifest-sli-input", "status": "success", "uid": "andes-cargo-manifest-sli", "url": "/d/andes-cargo-manifest-sli/andes-cargo-process-shipment-manifest-sli-input", "version": 1}This is the meaning of "reproducible artifact": anyone with access to this file and a running Grafana instance can recreate exactly the same panel, with the same uid, without relying on someone remembering to click in the right places in an interface.
Module 3's close
With OBSERVABILITY-RUNBOOK.md and dashboard.json written, this module delivers exactly what lesson 1 promised: not a complete instrumentation discipline, but the minimum pipeline that turns process-shipment-manifest's real invocations into the two numbers SLO.md's SLI needs — confirmed three times, through three paths (representative CloudWatch, real logs with jq, real Prometheus) — and demonstrated by running Module 2's calculator, with no code change at all, on that real data. You enter Module 4 with a real, documented telemetry pipeline — the foundation the burn rate alert is built on, with no need to rebuild any of these six lessons.
Common mistakes
Writing the runbook before having run the six previous lessons, "to get ahead" (documenting what you haven't verified yet). What happens: someone writes OBSERVABILITY-RUNBOOK.md based on what they expect the previous lessons did, without having really run docker compose up, the exporter, or the trace script. How to spot it: if you can't reproduce, by running the runbook's commands yourself, the same results lessons 3 through 6 already showed. How to fix it: the same standard SLO.md already demanded in Module 2 — every figure cited in a portfolio document must be traceable to a real run — applies here: every row of the "Where the data lives" table must correspond to something you really ran, not a reasonable assumption about how it should work.
Saving dashboard.json as a complete Grafana export, instead of the clean body used to create it (confusing "backup" with "reproducible artifact"). What happens: someone, exporting the dashboard from Grafana's UI, saves the complete JSON that includes internal metadata ("meta": {"canSave": true, "canEdit": true, ...}, "createdBy", instance timestamps), instead of the minimal body with "dashboard" and "overwrite" the API expects to create a new one. How to spot it: if your dashboard.json doesn't work when you pass it back to POST /api/dashboards/db on a different Grafana instance. How to fix it: this lesson's file is, deliberately, the request body you already used to create the dashboard — not an export of its metadata — precisely because that's the format that guarantees reproducibility on any Grafana instance, not just the one that generated it.
Treating the runbook's "Known limitations" as an apology, instead of operational information (underestimating the value of declaring what's representative). What happens: someone, writing the runbook's final section, treats it as a generic disclaimer ("some parts of this document may not be 100% up to date"), instead of naming exactly what's representative and why. How to spot it: if your "Known limitations" section doesn't mention LOCALSTACK_AUTH_TOKEN, or doesn't explicitly distinguish what really runs (Prometheus, Grafana, Jaeger) from what stays representative (CloudWatch). How to fix it: the same honesty discipline that governs this entire guide — declaring the exact technical reason for every representative piece, the moment it appears — applies with the same force inside a runbook: someone operating under pressure needs to know, precisely, what they can trust without double-checking and what still depends on a piece of infrastructure absent from this environment.
Exercises
Exercise 1 — Verify the complete runbook against your own run of lessons 3 through 6. For each row of the "Where the data lives" table, confirm you can reproduce, with your own commands, the state (representative or real) the runbook declares.
See solution
The five rows, verified: CloudWatch Metrics and CloudWatch Logs are representative because, in any environment with no LOCALSTACK_AUTH_TOKEN exported, attempting to run awslocal cloudwatch get-metric-statistics or awslocal logs filter-log-events would fail from being unable to connect to a LocalStack that never started — a negative verification, but a verification nonetheless. Prometheus, Jaeger, and Grafana are real because docker compose ps (lesson 6) shows all three containers in Running/Up state, and the direct HTTP queries (/-/ready, /api/health, /api/services) from lessons 5 and 6 responded with genuine data, not simulated. This exercise isn't a formality — it confirms the runbook, as a portfolio document, survives being audited line by line against real commands, the same standard SLO.md already set in Module 2.
Exercise 2 — Defend, against a technical interviewer, why this runbook explicitly separates "what's instrumented" from "where the data lives" from "what query reads it." An interviewer asks: "wouldn't a single document with all the commands, without three separate tables, be simpler?"
See solution
A complete answer: "The three questions are genuinely different, and someone operating under pressure needs them in that order. 'What's instrumented' is the design question — do we cover the three pillars that matter? — which almost never changes. 'Where the data lives' is the infrastructure question — is it in CloudWatch, in Prometheus, in Jaeger? — which can change if the team migrates tools, without the design question changing along with it. 'What query reads it' is the immediate operational question — the exact command someone copies and pastes at 3 AM — which depends directly on the previous answer. Merging all three into a single block of text would force rereading the whole document every time a single piece of infrastructure changed; separated, each table updates independently without touching the other two." The runbook's structure isn't decoration — it's the same separation of responsibilities lesson 2 of this module already established between metrics, logs, and traces, now applied to the documentation itself.
Exercise 3 — Explain why this project doesn't include an alert, even though the runbook mentions the data could be used to trigger one. Why does Module 3 stop at "measuring" and leave "alerting" for Module 4?
See solution
The same reason lesson 1 of this module already declared as a boundary: this module exists to answer one specific question — "do I have real data to calculate an SLI?" — not to build the complete incident-response system. An alert needs more than a data pipeline: it needs a threshold or burn rate rule (the Google SRE multi-window pattern Module 4 is going to formalize), a notification channel, and an explicit decision about when error-budget consumption is fast enough to justify waking someone up. None of those three pieces exists yet in this module — building them here, "while we're at it," would be exactly the kind of scope creep lesson 1 of this module warned about when drawing the boundary with monitoring-observability-guide. The runbook's "Known limitations" section declares it with the same honesty: this document measures, it doesn't alert — Module 4 is, deliberately, a different module.
Summary and next step
This module's final project wrote OBSERVABILITY-RUNBOOK.md — what's instrumented, where each piece of data lives, what exact query reads it, with complete honesty about what's representative and what's real — and saved observability/dashboard.json, lesson 6's Grafana panel, as a reproducible artifact, verified by recreating the same dashboard with the same uid from the file. You verified both documents with this whole guide's same deterministic discipline: 58 lines and 4 sections in the runbook, a "status": "success" identical to lesson 6's when recreating the dashboard from the file.
Before closing this module you should be able to: explain, from memory, the three questions that separate the runbook's three tables; name which of this module's data sources are real and which are representative, with each one's exact technical reason; and reproduce the Grafana panel from dashboard.json, with no dependency on the interface.
With this, Module 3 of sre-and-incident-response-guide is complete: observability as input to the SLI, never as a standalone discipline, with a real pipeline running over a fixed, deterministic batch, and the thread closed between what SLO.md defined in Module 2 and what this module measured with genuine data. Module 4 takes exactly this pipeline and builds this guide's first real alert: a rule that fires for the right reason — accelerated error-budget consumption, the same burn rate pattern from Module 2's lesson 6 — instead of an arbitrary threshold.
Resources
- This module, lessons 2 through 7 — the direct source for every row of
OBSERVABILITY-RUNBOOK.md's three tables. - This same repository, Module 2, lesson 8 (
08-project-andes-cargos-slo-md.md) —SLO.md, the document this runbook feeds with real data. - Grafana HTTP API — Dashboard — the official reference for the
dashboard.jsonformat this lesson saves. cloud-security-and-guardrails-guide(NIEVA), Module 1, project andfinops-and-cost-guardrails-guide(NIEVA), Module 1, project — the same short, verifiable portfolio document format, already used twice in this ecosystem.