Module 2: Slis Slos And The Error Budget
2. Choosing a good SLI: the four golden signals
Description
The previous lesson promised a tool. Before building it, a more basic question needs answering, one no calculator can solve for you: out of everything a system does, what's actually worth measuring? process-shipment-manifest has dozens of possible metrics — memory used, cold-start time, payload size, number of log lines per invocation — but an SLI isn't "any number the system produces." Google SRE solved this problem over a decade ago with a short, deliberately small list: the four golden signals. This lesson introduces them, quoted straight from the source, each with a concrete example on the Lambda you already know.
Connection to the module
Lesson 1 made clear a real SLI requires judgment, not just arithmetic. The four golden signals are that judgment, formalized: the list of serious candidates for any SLI, before deciding which one to use. Lesson 3 is going to take one of these four signals — errors — and turn it into this guide's first formal SLI definition, with the exact formula for good and valid events. This lesson is the filter that makes that choice possible with judgment, instead of at random.
Analogy: the dashboard of a car, not every sensor in the engine
A modern car easily has hundreds of sensors — oil temperature, pressure in each tire, battery voltage, dozens more. The dashboard, though, doesn't show you all of them: it shows you a handful of indicators designed to answer the question that actually matters while you drive — am I doing fine, or do I need to stop? Speedometer, tachometer, temperature gauge, fuel gauge. Four signals, not two hundred, chosen because each answers a different question and none is redundant with the others. Google SRE's four golden signals work the same way: they aren't the only metrics a system produces, they're the four that, together, answer almost any real question about whether a service is working well, with no need to look at the other two hundred sensors.
The four signals, quoted from the source
"If you can only measure four metrics of your user-facing system, focus on these four."
— Google SRE Book, Chapter 6 — Monitoring Distributed Systems
1. Latency
"The time it takes to service a request."
How long a request takes to resolve. The source adds an important nuance: you have to distinguish the latency of successful requests from the latency of requests that fail — a request that fails instantly (a fast validation error) and one that fails after exhausting a full timeout are, for latency purposes, two completely different stories, even though both count as "error" for the next signal.
On process-shipment-manifest: the Timeout: 10 that lesson 4 of Module 1 already identified in the Lambda's real configuration is, literally, this signal's hard limit. This Lambda's latency is how long it takes to process a complete manifest — parsing the file, validating the data, writing to Shipments — from the moment S3 fires the event until the function finishes. An invocation that takes 9 seconds technically "worked," but it's dangerously close to the limit; one that takes 11 gets force-killed, no matter how close it was to finishing. Measuring real latency, not just whether there was a timeout or not, is what would let you catch that degradation before it crosses the line.
2. Traffic
"A measure of how much demand is being placed on your system, measured in a high-level system-specific metric."
How much demand the system receives, in whatever unit makes sense for that specific system — HTTP requests per second for an API, transactions per second for a database system. The source is explicit that the right unit depends on the system, there's no single universal traffic metric.
On process-shipment-manifest: the natural unit is invocations per day — how many manifests enter the system in a time window. This is, exactly, the "valid events" column you're going to see in lesson 4's dataset: each row of the fixed 30-day dataset is, at bottom, a daily traffic measurement. The ReservedConcurrentExecutions: 5 ceiling Module 1 already identified turns this signal into something more urgent than "how much demand is there" — it's also the answer to "how much demand can this system absorb before it starts throttling requests?"
3. Errors
"The rate of requests that fail, either explicitly (e.g., HTTP 500s), implicitly (for example, an HTTP 200 success response, but coupled with the wrong content), or by policy (for example, if you've agreed to return a 400 error state)."
The rate of requests that fail — and the source insists on three distinct ways of failing, not just one. An explicit error is easy to detect (an error status code). An implicit error is more dangerous precisely because it doesn't look like an error: the system responds "success," but with the wrong content.
On process-shipment-manifest: an explicit error is an unhandled exception during processing — an invocation that ends in a failure state, visible in the logs and in CloudWatch's Errors metric. An implicit error is subtler and is exactly the kind of risk lesson 4 of Module 1 already named without resolving: an invocation that finishes "successfully" (no exception, no timeout) but writes an incomplete or corrupted record to Shipments — the Lambda reports success, but the real data is wrong. This guide picks this distinction back up with precision in lesson 3, when deciding exactly what counts as "good" for this Lambda's availability SLI.
4. Saturation
"How 'full' your service is. A measure of your system fraction, emphasizing the resources that are most constrained (e.g., in a memory-constrained system, show memory; in an I/O-constrained system, show I/O)."
How full the system is — not in general, but specifically in the resource that most constrains its capacity. The source clarifies that saturation isn't just "is it at 100%?": for complex systems, the point where performance starts degrading usually arrives before 100% utilization, so a good saturation indicator includes a notion of the point beyond which degradation starts to be noticeable.
On process-shipment-manifest: this Lambda's most constrained resource, per its own configuration, isn't memory (MemorySize: 128 is generous for a manifest processor) — it's concurrency. ReservedConcurrentExecutions: 5 means this system's saturation is measured as "how many of the 5 concurrency slots are occupied right now?" The sixth simultaneous invocation isn't a code error or a latency failure — it's pure saturation: the system is, literally, full, and the request gets throttled before it even attempts to run.
The four signals, applied to process-shipment-manifest: the complete table
| Signal | What it measures, per the source | On process-shipment-manifest | Configuration data you already know (Module 1) |
|---|---|---|---|
| Latency | Time to resolve a request | Seconds from the S3 trigger until the invocation finishes | Timeout: 10 |
| Traffic | Demand on the system | Invocations per day (the base column of lesson 4's dataset) | ReservedConcurrentExecutions: 5 (the absorbable demand ceiling) |
| Errors | Rate of requests that fail (explicit, implicit, or by policy) | Unhandled exceptions (explicit); corrupted writes to Shipments (implicit) | No confirmed DLQ on this path (Module 1, lesson 4) |
| Saturation | How full the most constrained resource is | Concurrency slots occupied, out of 5 available | ReservedConcurrentExecutions: 5 |
Why this guide picks errors, not all four, for the availability SLI
The four signals are serious candidates — that doesn't mean all of them automatically become the SLI this guide is going to formally define in lesson 3. The reason is about focus, not hierarchy: RELIABILITY-CHARTER.md's central question is "what does 'reliable' mean for this system?", and for an asynchronous manifest processor, "reliable" translates, above all, into "did the manifest get processed correctly?" — the errors signal, measured with the exact formula of good events ÷ valid events, is the one that answers that question most directly. Latency, traffic, and saturation don't disappear from this guide: latency reappears as a complementary signal in Module 3 (measured with real CloudWatch data); saturation is, literally, one of the six risks lesson 4 of Module 1 already named, still unresolved. But Andes Cargo's primary SLI — the one SLO.md is going to set in lesson 8 of this module — is built on errors, the signal that most directly answers the question the business cares about.
Common mistakes
Treating the four signals as if all of them had to become formal SLIs (over-engineering). What happens: someone, after this lesson, tries to define four separate SLIs for process-shipment-manifest — one per signal — before moving on. How to spot it: if your plan for lesson 3 is to write four SLI formulas instead of one. How to fix it: the Google SRE source itself presents the four signals as candidates to consider, not as an obligation to instrument all four with the same rigor. This guide picks errors as Andes Cargo's primary SLI for an explicit business reason (above); the other three stay relevant — they show up in Module 3 as observability data — but not all of them need to become a formal SLI with its own SLO.
Confusing "traffic" with "SLI" (wrong category). What happens: someone proposes "the number of daily invocations" as an SLI in itself. How to spot it: if your SLI proposal has no notion of "good" versus "bad" — just a count. How to fix it: traffic is, almost always, an SLI's denominator (the "valid events" from the formula lesson 3 is going to formalize), not the SLI itself. An SLI needs a ratio — something divided by something — and "how many invocations there were" is only one of the two parts of that ratio.
Ignoring saturation because "the system has never run out of memory" (looking at the wrong resource). What happens: someone dismisses saturation as a relevant signal for process-shipment-manifest, reasoning that memory (128 MB) has never been a problem. How to spot it: if your saturation analysis only considers memory or CPU. How to fix it: the source is explicit about focusing on the most constrained resource, not any resource available — for this specific Lambda, that resource is reserved concurrency (5 slots), not memory. Measuring memory when the real bottleneck is concurrency is measuring the wrong signal with perfect precision.
Exercises
Exercise 1 — Classify a hypothetical symptom under the right signal. An Andes Cargo customer reports that, for an hour, no new manifests were processed, even though the Lambda still showed as "active" in the console. Investigating, you find the five concurrency slots were continuously occupied by invocations taking 9.8 seconds each, dangerously close to the Timeout: 10. Which of the four golden signals (or signals) explains this symptom, and why might more than one apply?
See solution
Two signals apply at once, and it's worth naming both: saturation (the five concurrency slots were continuously occupied, the system literally full) and latency (9.8 seconds per invocation, dangerously close to the 10-second limit, is the root cause of why the slots took so long to free up). The relationship between the two is direct: high latency extends how long each invocation holds a concurrency slot, which makes saturation arrive faster with the same traffic volume. This is exactly the kind of connection between signals that makes the four, together, tell a more complete story than any one alone — saturation on its own would say "the system is full"; adding latency explains why it got full.
Exercise 2 — Explain why an "implicit" error is more dangerous than an "explicit" one for process-shipment-manifest, with a concrete example. Use the definition quoted in this lesson.
See solution
An explicit error — an unhandled exception, a Timeout — leaves automatic evidence: it shows up in the logs, it increments CloudWatch's Errors metric, it's visible without anyone having to look carefully for it. An implicit error, per the definition quoted ("an HTTP 200 success response, but coupled with the wrong content"), is dangerous precisely because it generates none of those signals — the Lambda finishes, reports success, the Errors metric doesn't move. A concrete example for process-shipment-manifest: an invocation that processes shipment 4471's manifest but, due to a field-mapping error, writes that shipment's status into the Shipments record for shipment 4472 — the Lambda "worked" from CloudWatch's perspective, but the real data for two different shipments is now corrupted. This kind of error only gets caught with content validation, never with a technical failure-rate metric, which is exactly the distinction lesson 3 of this module has to resolve when deciding what counts as "good."
Exercise 3 — Argue why this guide picks errors, not saturation, as Andes Cargo's primary SLI, even though saturation is a real, already-identified risk. A teammate suggests that, since saturation (concurrency) is "the most visible risk" from Module 1, it should be the basis for the main SLI instead of errors.
See solution
A risk's visibility in an inventory (Module 1, lesson 4) isn't the same as its direct relevance to RELIABILITY-CHARTER.md's central question: "what does 'reliable' mean for this system?" For an Andes Cargo customer, what matters isn't whether the system was "full" at some point — that's a possible root cause — it's whether their manifest got processed correctly or not, which is exactly what the errors signal measures directly. Saturation is, in fact, one of the most likely causes of an error (an invocation throttled for lack of concurrency ends up, for the customer, being a failed invocation) — but the SLI is defined on the symptom that matters to the end user (did it work?), not on the internal technical cause (why didn't it work?). This distinction — the SLI measures the user's experience, not the internal mechanism — is the same one that separates "what an SLI measures" from "what a postmortem explains," and the guide applies it consistently starting with this lesson.
Summary and next step
In this lesson you learned Google SRE's four golden signals — latency, traffic, errors, saturation — each quoted from the original source and applied with a concrete example on process-shipment-manifest, using real configuration data you already knew from Module 1 (Timeout: 10, ReservedConcurrentExecutions: 5). You also saw why this guide picks errors as the basis for Andes Cargo's primary SLI: not because the other three signals don't matter, but because errors is the one that most directly answers the business question RELIABILITY-CHARTER.md raised.
Before moving on you should be able to: name the four signals from memory, with their exact quote; explain the difference between an explicit and an implicit error, with your own example from process-shipment-manifest; and defend why errors, not saturation or latency, is the signal chosen for this guide's primary SLI.
Lesson 3 takes the errors signal and turns it, in writing, into this guide's first formal SLI definition: what exactly counts as "good," what exactly counts as "valid" — the judgment call no calculator can make for you.
Resources
- Google SRE Book, Chapter 6 — Monitoring Distributed Systems — the exact source for the four golden signals, quoted in this lesson.
- This same repository, Module 1, lesson 4 (
04-hands-on-reading-andes-cargo-like-an-sre.md) — the realprocess-shipment-manifestinventory (Timeout: 10,ReservedConcurrentExecutions: 5) this lesson reuses for each signal. - Google SRE Book, Chapter 4 — Service Level Objectives — the SLI/SLO framework lesson 3 of this module formalizes with the errors signal.