Module 3: Observability As Sli Input

1. Introduction: observability in service of a question, not a dashboard

Description

Module 2 closed with SLO.md: an SLI defined with precision, a 99.9% monthly SLO chosen with evidence, and a 43.2-minute budget measured — but measured over a fixed dataset, committed by hand, not over data coming out of process-shipment-manifest at this very moment. SLO.md was explicit about that limitation, in its own "Consequences" section: "not a live number [...] but the evidence this SLO was chosen with, not guessed at." This module exists to resolve exactly that sentence: build the minimum pipeline that turns real invocations of the inherited Lambda into the same two numbers compute_sli() needs — valid events, good events — using observability's three classic pillars: metrics, logs, and traces.

The key word in that sentence is minimum. This module isn't an instrumentation course. You aren't going to learn what a counter is versus a histogram, or instrument an application from scratch, or design a complete organization-wide dashboard. You're going to learn exactly what it takes for one very specific question — is process-shipment-manifest's SLI being met? — to have an answer backed by real data, not a sample dataset. Everything this module instruments serves that question; nothing else.

Connection to the module

This module's eight lessons follow the same pattern you already used in Module 2: first the framework (lesson 2, the three pillars through an SRE lens), then the real execution, pillar by pillar (lessons 3 through 6: metrics, logs, traces, and the Prometheus/Grafana stack the market demands), and at the end the thread's close (lesson 7: Module 2's calculator running, for the first time, with real telemetry from this module instead of the fixed dataset). Lesson 8 documents the complete pipeline as Andes Cargo's first operational observability artifact.

The boundary this module respects: instrumenting is a means, not an end

This ecosystem already has a guide dedicated to observability as a complete discipline: monitoring-observability-guide. That guide — not this one — is where you learn what a counter is versus a histogram, how to instrument an application from scratch, and how to design organization-wide dashboards in depth. aws-serverless-and-containers-guide/DISENO.md already delegated that discipline there, explicitly, before this guide even existed.

This module does something different and narrower: it takes the exact question SLO.md left open — is there real data behind this SLI? — and instruments the bare minimum needed to answer it. Every lesson in this module, with no exception, can be summarized with the same question: does this let me calculate a real SLI? Never: "how do you instrument an application in general?" That broader question remains, deliberately, monitoring-observability-guide's territory — this module doesn't answer it, and shouldn't.

The difference shows up in depth, not in the tools: you're going to use Prometheus, Grafana, CloudWatch, OpenTelemetry, and Jaeger — the same tools a full observability course would use — but you're going to use them to produce exactly two numbers (valid events, good events) and to feed them to Module 2's calculator. No panel in this module exists "because it looks good on a dashboard"; every one exists because it feeds, directly or indirectly, compute_sli().

This module's map: the 8 lessons

   MODULE 3 — OBSERVABILITY AS SLI INPUT
   from SLO.md with a fixed dataset to SLO.md measured with real data

   M3.1  Introduction (this lesson)                the boundary, the module's single question
   M3.2  The three pillars, with an SRE lens         which question each one answers
   M3.3  Real metrics from the inherited Lambda       REPRESENTATIVE: awslocal cloudwatch
   M3.4  Real logs with jq                            REPRESENTATIVE + REAL: jq over real logs
   M3.5  Traces with OpenTelemetry and Jaeger          REAL: OTel -> Jaeger v2, running
   M3.6  Prometheus and Grafana, the market's stack     REAL: docker compose, real panel
   M3.7  From raw telemetry to a measured SLI            REAL: the M2.4 calculator, real data
   M3.8  Project: the observability-to-SLI pipeline       REAL: runbook + saved panel

#	Lesson	What it builds
1	Introduction (this one)	The boundary with `monitoring-observability-guide`; the module's single question
2	The three pillars, with an SRE lens	Which question each pillar answers when the question is "is my SLI being met?"
3	Real metrics from the inherited Lambda	Representative: `awslocal cloudwatch get-metric-statistics` over a fixed batch of 20 invocations
4	Real logs with `jq`	Representative (awslocal) + real (`jq`) — extracting the 3 failed invocations by `requestId`
5	Traces with OpenTelemetry and Jaeger	Real: two traces, `upload → Lambda → DynamoDB`, running against local Jaeger v2
6	Prometheus and Grafana, the market's stack	Real: `docker compose up`, a Grafana panel with the same success/error metric
7	From raw telemetry to a measured SLI	Real: `error_budget_calculator.py` runs with this module's data, not the M2.4 dataset
8	Project: the observability-to-SLI pipeline	Real: the short runbook + the Grafana panel saved as `dashboard.json`

This module's deterministic thread: one batch, three lenses

Unlike Module 2 — where each lesson ran over a different dataset (30 days of traffic, then a bad week) — this module runs over a single, fixed batch of 20 process-shipment-manifest invocations, with 3 deliberately malformed at fixed positions (5, 12, and 17), never chosen with random. That same batch shows up read through three different instruments:

Lesson 3 reads it as an aggregate count (AWS/Lambda/Invocations/Errors): 20 invocations, 3 errors.
Lesson 4 reads it as per-invocation detail (logs): which requestId failed, and why — the same validation error you already know from aws-serverless-and-containers-guide, not one invented for this guide.
Lesson 5 reads it as one specific invocation, followed end to end (a trace): number 17, with the exact point where the upload → Lambda → DynamoDB flow breaks.

Three different questions about the same real event, not three different events — exactly the idea lesson 2 develops with this module's complete analogy.

This module's honesty: what really runs, what stays representative, and why

This module has the most nuanced real/representative split in this whole guide, and it's worth seeing in full before the first technical lesson:

Tool	This module's status	Exact technical reason
`awslocal cloudwatch get-metric-statistics` (lesson 3)	Representative	With no `LOCALSTACK_AUTH_TOKEN` exported in this writing environment, the LocalStack container doesn't start. CloudWatch is confirmed on LocalStack's Hobby plan — the output you'll read is what that command would produce against the real infrastructure, reconstructed field by field, never invented.
`awslocal logs filter-log-events` (lesson 4)	Representative, same reason	Same as above: the command exists, it's correct, and it would run against LocalStack Hobby with the token exported.
`jq` (lesson 4)	Real	`jq` doesn't depend on LocalStack. It runs against a sample logs JSON file — the same format the `awslocal` command above would produce — and produces this lesson's literal output, verified by running the real binary.
OpenTelemetry SDK + Jaeger v2 (lesson 5)	Real	`instrument_manifest_flow.py` runs with `python3`, sends spans over OTLP/HTTP to a real Jaeger v2 container, running with `docker run`. The trace you'll see in the Jaeger UI is the one that script produced, verified in this environment.
Prometheus + Grafana (lesson 6)	Real	`docker compose up` brings up both containers for real; the Grafana panel queries Prometheus with real PromQL, over a real metric exposed by a real Python exporter.
`error_budget_calculator.py` (lesson 7)	Real	The same calculator from Module 2, running with `python3`, this time with this module's numbers.
AWS X-Ray (named in lesson 5)	Representative, named, not run	"Included in Plans: Ultimate" — absent from LocalStack's Hobby plan. This is the exact reason this lesson uses OpenTelemetry + Jaeger instead of X-Ray.

The rule that governs this whole guide holds with no exception: if a command appears in a lesson, it ran to write it. What didn't really run — the two awslocal commands — gets labeled at the exact moment it appears, with the precise technical reason, never a generic "this should work."

Common mistakes

Expecting this module to teach observability "in general" (scope confusion). What happens: someone arrives at this module expecting to learn, for example, how to instrument any application with OpenTelemetry, or how to design a Grafana dashboard from scratch for any system. How to spot it: if your question after finishing a lesson in this module is "but how would I do this for ANOTHER system, with other metrics?" instead of "does this feed process-shipment-manifest's SLI?" How to fix it: that broader scope is, deliberately, monitoring-observability-guide's territory — a complete guide dedicated to that question. This module instruments the bare minimum for a specific case, with an SLI already defined beforehand; it's not a general course on the discipline.

Treating this module's two awslocal commands as if they were less real than the rest (underestimating the representative label). What happens: someone reads "representative" and assumes that part of the lesson is less trustworthy, or can be skipped. How to spot it: if your plan is "let me get to the real parts, I'll skip this." How to fix it: representative doesn't mean invented — it means reconstructed field by field from already-confirmed behavior (CloudWatch on LocalStack's Hobby plan, official AWS documentation), with the only variable being that this specific writing environment doesn't have the LocalStack token exported. If you run these lessons with your own LOCALSTACK_AUTH_TOKEN, you should get equivalent results.

Thinking lessons 3, 4, and 5 measure three different events (losing the deterministic thread). What happens: someone reaches lesson 5 thinking the Jaeger trace is of a new event, unrelated to lesson 3's counts or the requestId extracted in lesson 4. How to spot it: if you can't say which position in the 20-invocation batch corresponds to the error trace lesson 5 shows in Jaeger. How to fix it: all three lessons read the same fixed 20-invocation batch — lesson 5's error trace is, specifically, invocation number 17, the same one lesson 4 extracts by requestId with jq. It's a single batch, read through three instruments, not three different batches.

Exercises

Exercise 1 — Explain, without looking back, what exact sentence from SLO.md (Module 2, lesson 8) this module exists to resolve. Cite the section of SLO.md where it appears, and explain in one sentence what changes between that section and this module's end.

See solution

The sentence lives in SLO.md's "Consequences" section: "not a live number [...] but the evidence this SLO was chosen with, not guessed at," referring to the 4.23 remaining budget minutes measured on Module 2 lesson 4's fixed dataset. What changes by this module's end: lesson 7 runs the exact same calculator (compute_sli(), budget_report()) with data coming from a real telemetry source — Prometheus, fed by real invocations from this module's batch — instead of the hand-committed dataset. The SLI stays the same one defined in SLO.md; what changes is where the numbers fed into the formula come from.

Exercise 2 — Classify a hypothetical task: does it belong to this module, or to monitoring-observability-guide? A teammate proposes instrumenting process-shipment-manifest with a latency histogram by percentile (p50, p95, p99), thinking of a general Lambda performance dashboard, with no connection to any specific SLI yet. Does that task fall within this module's scope?

See solution

It doesn't fall within this module's scope, as described. The question that governs every lesson in this module is "does this let me calculate a real SLI?" — and a latency histogram by percentile, with no declared connection to a specific SLI or SLO, is exactly the kind of general instrumentation monitoring-observability-guide covers. If the same proposal were reframed as "we need p99 latency because this module's SLO depends on no invocation exceeding Timeout: 10," then it would fall within scope — because now the metric serves a concrete SRE question, not a general performance dashboard. The difference isn't the tool (a histogram is a legitimate tool in both cases); it's whether an SLI question exists behind it or not.

Exercise 3 — Without reading lesson 2 yet, predict which question each of the three pillars (metrics, logs, traces) is going to answer when the central question is "is my SLI being met?" Write one sentence per pillar, using what you already know from Module 2's SLI definition (good events ÷ valid events).

See solution

There's no single correct answer before reading lesson 2, but a reasonable prediction, using the SLI formula: metrics would answer "how many valid events were there, and how many of those were good?" — the aggregate count that directly feeds compute_sli()'s numerator and denominator. Logs would answer "which ones, specifically, were the bad events, and why?" — the detail aggregate metrics don't give, needed to act, not just to measure. Traces would answer "at what exact point in the flow did a specific invocation break?" — the single-invocation, end-to-end view, useful for diagnosing a specific failure, not for calculating the month's SLI. Lesson 2 formalizes this same intuition with this module's complete analogy.

Summary and next step

This lesson installed the boundary that governs the seven lessons that follow: this module doesn't teach observability as a general discipline — that's monitoring-observability-guide — it instruments the bare minimum needed for one specific question, the one SLO.md left open, to have an answer backed by real data. You saw the map of the 8 lessons, the deterministic thread connecting them (a single 20-invocation batch, read through three different instruments), and this module's complete honesty table: what really runs (jq, OpenTelemetry, Jaeger, Prometheus, Grafana, the calculator) and what stays representative, with its exact technical reason (the two awslocal commands).

Before moving on you should be able to: explain in one sentence the difference between "instrumenting as an end" and "instrumenting as a means," citing the exact sentence from SLO.md this module resolves; name the 8 lessons in order; and explain why lessons 3, 4, and 5 aren't three different events, but three lenses on the same fixed batch.

Lesson 2 formalizes the three pillars — metrics, logs, traces — with this module's complete analogy: three ways of investigating why a train arrived late.

Resources

This same repository, Module 2, lesson 8 (08-project-andes-cargos-slo-md.md) — SLO.md, with the "Consequences" section this module resolves.
aws-serverless-and-containers-guide (NIEVA), Module 2 — the version of process-shipment-manifest (handler.py, validate_manifest()) this module reads with instrumentation, without rewriting it.
monitoring-observability-guide (NIEVA) — the sibling guide that owns observability as a general discipline; the boundary declared in this lesson.
Google SRE Book, Chapter 6 — Monitoring Distributed Systems — the monitoring framework lesson 2 applies through an SRE lens.
LocalStack Docs — CloudWatch — confirmation of the Hobby plan, the source for lessons 3 and 4's "representative" label.

← Previous8. Project: Andes Cargo's `SLO.md`Next →2. The three pillars, with an SRE lens