Module 3: Observability As Sli Input
2. The three pillars, with an SRE lens
Description
Metrics, logs, and traces are, almost certainly, three words you already heard before this guide — they're, literally, "the three pillars of observability," a phrase repeated in nearly every technical conference of the last decade. This lesson doesn't present them as new concepts. It presents them with a specific constraint most of those conferences don't apply: each one, seen only through the question this module exists to answer — is process-shipment-manifest's SLI being met? That constraint changes what matters about each pillar and what doesn't, and it's exactly the lens that separates this module from a general observability course.
Connection to the module
Lesson 1 promised this analogy; this lesson develops it in full, and lessons 3, 4, and 5 execute it for real, pillar by pillar, over the same fixed 20-invocation batch you already know from the module's map. Everything that follows in this module is a concrete instance of the distinction this lesson establishes.
Analogy: three ways of investigating why a train arrived late
A passenger train arrives 40 minutes late at its destination. The railway company needs to know what happened, and it has three completely different sources of information, each useful for a different question.
The punctuality board shows, for every train on the network, the scheduled arrival time, the real arrival time, and the difference — one number per train, aggregated across the whole day's operation. With that board, someone can quickly answer: "how many trains were late today? what's the network's punctuality rate this week?" What the board can't answer is why a specific train was late — it only says it was late, and by how much.
The engineer's logbook is the written record, train by train, of every relevant event during the trip: "14:02 — unscheduled stop at km 340 due to red signal," "14:15 — resumed movement," "14:40 — reduced speed due to trackwork." With that logbook, someone can answer the question the board left open: this specific train was late for two documented reasons, each with its exact time. The logbook has the detail the board never had — but it's only useful if you already know which specific train you want the story for.
Following a specific car along the whole route — with a person or a dedicated sensor tracking a single car from origin to destination — is the third way, the most expensive and the least often needed: not just knowing there was an unscheduled stop, but seeing exactly at what point on the route, within which segment, with what other car coupled, each event happened — an end-to-end view of an individual case, useful when the logbook alone isn't enough to understand a specific incident's full sequence.
These three ways of investigating are, exactly, metrics, logs, and traces — and the reason none replaces the other two is the same reason a serious railway company needs all three.
Metrics: process-shipment-manifest's punctuality board
A metric is a number aggregated over a time window — a count, a sum, an average — with no detail on any individual event. AWS/Lambda/Invocations and AWS/Lambda/Errors, the two metrics lesson 3 is going to read with awslocal cloudwatch get-metric-statistics, are exactly this: how many invocations there were in a window, how many of those ended in error. Nothing more, and nothing less.
The question it answers, when the central question is "is my SLI being met?": metrics are, literally, compute_sli()'s two inputs — total_valid comes from Invocations, and total_good is derived from Invocations minus Errors. No other data source in this module feeds the SLI formula this directly. If you had to pick a single pillar to calculate SLO.md's monthly SLI, it would be this one.
What it doesn't answer: why the 3 invocations failed. AWS/Lambda/Errors with Sum: 3.0 tells you something failed three times — it doesn't tell you whether it was the same error three times, three different errors, or in which specific invocation each one happened. That "why" is, exactly, the question the next pillar answers.
Logs: each invocation's engineer's logbook
A log is a record with detail on a specific event, almost always with a timestamp and enough context to reconstruct what happened — in process-shipment-manifest's case, every invocation writes its own sequence of lines to /aws/lambda/process-shipment-manifest: when it started (START RequestId: ...), what the code did (the function's print() calls), when it finished (END RequestId: ...), and a cost and status summary (REPORT RequestId: ...).
The question it answers: of the 3 invocations Errors flagged as failed, which ones, exactly — with their requestId — and why, with the real message validate_manifest() produced? Lesson 4 extracts exactly this, with jq, over real logs from the batch's three failed invocations.
What it doesn't answer: at what exact point in a multi-step flow — S3 upload, Lambda processing, DynamoDB write — the break happened, when that flow involves more than one service. A process-shipment-manifest log tells you the function failed; it doesn't show you, in a single view, the time relationship between the S3 upload that triggered it and the Shipments write attempt that never happened. That "at what point in the flow" is the third pillar's question.
Traces: following a specific invocation through the whole flow
A trace is the record of an individual operation, followed across every service involved in it, with the time and hierarchical relationship between each step — in OpenTelemetry, each step is a span, and the related spans of a single operation form a trace. Lesson 5 builds exactly this for process-shipment-manifest: a trace with three spans — shipment-manifest-upload (the S3 upload), process-shipment-manifest (the Lambda invocation), dynamodb-put-item (the write to Shipments) — for the successful path, and a trace with two spans — where the third never opens — for a failing invocation.
The question it answers: for one specific invocation — number 17 of the batch, the same one lesson 4 already identified by requestId with logs — at what exact step of the flow did it break, and which earlier steps did complete? Lesson 5's error trace precisely shows that the shipment-manifest-upload span did happen, that process-shipment-manifest ended in ERROR state, and that no DynamoDB span even opened — visual proof the flow stopped exactly at validation, before any write attempt.
What it doesn't answer: how many invocations, in total, had the same problem. A trace is, by design, the view of one invocation — or a handful of hand-picked ones — never an aggregate. For that, the cycle returns to the first pillar: metrics.
The three pillars, applied to the same batch: the complete table
| Pillar | Question it answers (SRE lens) | Tool in this module | Feeds compute_sli() |
|---|---|---|---|
| Metrics | How many valid events were there, and how many were good? | awslocal cloudwatch get-metric-statistics (lesson 3) | Directly — total_valid, total_good |
| Logs | Which ones, specifically, were the bad events, and why? | awslocal logs filter-log-events + jq (lesson 4) | Indirectly — confirms and explains what the metrics already counted |
| Traces | At what exact step of the flow did a specific invocation break? | OpenTelemetry + Jaeger (lesson 5) | Indirectly — diagnoses, doesn't count |
Notice the last column: only metrics feed the SLI formula directly. Logs and traces are essential for understanding an SLI that's being missed — but the number SLO.md cites, 99.9098% or whatever it is, gets calculated with metrics, not logs or traces. This hierarchy — metrics to measure, logs and traces to diagnose — is the exact reason lesson 3 of this module comes first.
Why all three, and not just the first
A reasonable question, after the table above: if only metrics feed the SLI directly, why build logs and traces in this module at all? The answer is the same one you already saw in Module 2, lesson 2, when choosing errors over the other three golden signals: an SLI tells you that something is wrong, never why. An SLI of 85% on this module's batch — the exact number lesson 7 is going to calculate — is alarming, but on its own it doesn't say whether the problem is a real bug in validate_manifest(), an unannounced change in the format of incoming manifests, or three isolated, unrelated cases. That question — the step that separates "measuring" from "acting" — is exactly what logs and traces answer, and it's exactly why an SRE never settles for an SLI with no ability to investigate behind it.
Common mistakes
Treating the three pillars as if they were interchangeable (assuming "more observability" is always better). What happens: someone, after this lesson, concludes that with traces you no longer need any logs, or that with metrics you no longer need any traces. How to spot it: if your plan is to instrument a single pillar "because it's the most complete." How to fix it: this lesson's table is explicit that each pillar answers a question the other two don't — aggregate (metrics), single-event detail (logs), an operation's complete flow (traces). None substitutes for the other two; each one covers a question the others leave unanswered.
Confusing "having traces" with "having an SLI" (overestimating what traces contribute to the formula). What happens: someone, after seeing a real trace in Jaeger in lesson 5, assumes they already have everything needed to calculate the month's SLI. How to spot it: if your plan for lesson 7 is to use Jaeger data, instead of metrics data, as compute_sli()'s input. How to fix it: this module's traces deliberately cover only two invocations out of the 20-invocation batch — one successful, one failed — hand-picked to illustrate the complete flow. An SLI needs the total count of valid and good events, which only aggregate metrics give without having to instrument (and pay the cost of) a trace for each of the thousands of real invocations process-shipment-manifest would receive in production.
Thinking this module instruments the three pillars "because you always have to" (losing the question that justifies them). What happens: someone treats this module as a generic checklist — "every serious system needs metrics, logs, and traces" — without connecting each one to this module's specific SLI question. How to spot it: if you can't explain, for each of the three pillars, what specific SRE question it answers about process-shipment-manifest. How to fix it: lesson 1 already made it explicit — this module's single question is "does this let me calculate a real SLI?" The three pillars belong here because, together, they answer that whole question (measuring, and being able to explain what was measured), not because a generic checklist demands them.
Exercises
Exercise 1 — Apply the train analogy to a new scenario, without using the technical names yet. An Andes Cargo customer reports their shipment 4473 never got registered in the system. What question, in the right order, would you ask first with the "board," then with the "logbook," and last "following the car"? Explain why that order, and not another.
See solution
First, the board (metrics): "was there any error in process-shipment-manifest in the time window when shipment 4473's manifest was uploaded?" — an aggregate question, fast to answer. If the answer is yes, second, the logbook (logs): "which specific invocation failed, with what requestId, and what error message did it leave?" — now you already know there was at least one error, and you need the detail of which one. Only if the log isn't enough to understand the full sequence — for example, if the error happened during the DynamoDB write after a successful validation, and you need to see the time relationship between the two steps — the third step, following the car (traces): an end-to-end view of that specific invocation. The order matters because each step is more expensive and more specific than the last — it wouldn't make sense to open a trace for a specific invocation before confirming, with the aggregate metric, that there was some error worth investigating.
Exercise 2 — Explain why AWS/Lambda/Errors with Sum: 3.0, on its own, doesn't tell you whether the three errors were the same problem or three different ones. Which specific pillar resolves that ambiguity, and how?
See solution
Errors is an aggregate count — a sum over a time window — with no detail on what caused each individual error; three failed invocations with the Errors: 3.0 metric could, in principle, be three instances of the same bug, three different bugs, or any combination. Logs resolve this ambiguity exactly because every log line from a failed invocation (the real print(f"Invalid manifest {key}: {errors}") from validate_manifest()) carries that invocation's specific error message — lesson 4 of this module is going to show that, out of the 20-invocation batch, the three failed invocations have three different validation reasons (missing weightKg field, missing carrier field, non-numeric weightKg), information no aggregate metric could have revealed.
Exercise 3 — Defend, against a skeptical teammate, why this module doesn't stop after building just lesson 3's metrics. Your teammate argues: "metrics already feed compute_sli() directly, according to this lesson's table — why spend two more lessons on logs and traces?"
See solution
A complete answer: "It's true that only metrics feed compute_sli() directly — this very lesson says so — but an SLI that can only be measured, with no way to investigate it when it's missed, is an alarm with no path to action. If a month's SLI drops below the SLO, the team's immediate question is going to be 'why, and what do we do?' — and no aggregate number answers that question; logs answer it (which specific error, in which invocation) and, when the log isn't enough, traces answer it (at which exact step of the flow). Stopping at metrics would be like installing a smoke alarm with no plan for what to do when it goes off: it measures the problem, but it doesn't help solve it." This lesson's table already distinguishes "measuring" (metrics) from "diagnosing" (logs and traces) — a serious observability system needs both capabilities, not just the first.
Summary and next step
This lesson introduced observability's three pillars — metrics, logs, traces — with this module's complete analogy: the punctuality board, the engineer's logbook, and following a specific car along the whole route. You saw, precisely, what question each one answers when the central question is "is my SLI being met?": metrics feed compute_sli() directly (valid events, good events); logs explain, invocation by invocation, which ones failed and why; traces show, for a specific invocation, at what exact step of the upload → Lambda → DynamoDB flow it broke.
Before moving on you should be able to: recite the complete train analogy, with the three pillars correctly assigned; explain why only metrics feed the SLI formula directly; and argue why an SLI with no logs or traces behind it is a measurement with no path to diagnosis.
Lesson 3 executes the first pillar for real: the same 20-invocation batch, read as an aggregate count with awslocal cloudwatch get-metric-statistics over AWS/Lambda/Invocations and AWS/Lambda/Errors.
Resources
- Google SRE Book, Chapter 6 — Monitoring Distributed Systems — the Google SRE monitoring framework this lesson applies through an SLI lens, already cited in Module 2 for the four golden signals.
- OpenTelemetry — Observability Primer — the official definition of the three pillars (metrics, logs, traces) this lesson adapts with the train analogy.
- This same repository, Module 2, lesson 4 (
04-hands-on-the-error-budget-calculator.md) —compute_sli(), the function this module's lesson 3 metrics feed directly. aws-serverless-and-containers-guide(NIEVA), Module 2 —validate_manifest(), the real source of the three different error messages this module's lesson 4 extracts withjq.