Module 2: Slis Slos And The Error Budget

3. Hands-on: your first SLI definition

Description

The previous lesson picked errors as the signal Andes Cargo's primary SLI is going to measure. This lesson does the real work of turning that choice into a precise definition — in writing, with explicit judgment, decided case by case — before a single line of the calculator's code exists. This is the part of the process that looks least like programming and most like a product exercise: deciding, with evidence and not for convenience, what exactly counts as a "good event" and what exactly counts as a "valid event" for process-shipment-manifest.

Connection to the module

Lesson 4's calculator is going to receive a dataset with two numbers per day: valid events and good events. Those two numbers aren't an implementation detail — they're the complete SLI definition, summarized into two columns. If this lesson gets skipped or done carelessly, lesson 4 runs a perfectly correct script over data that doesn't mean what it should mean. This is, literally, the lesson that makes the rest of the module measure something real.

The exact formula, with its source

The SLI formula this module uses isn't "good events ÷ total events" — it's more precise than that, and the precision matters:

"The SLI Equation: The proportion of valid events that were good."

"Events can be prevented from counting against an error budget either by including them in the numerator or by excluding them from the denominator. The former is achieved by classifying some events good, the latter by classifying some events invalid."

— Google — The Art of SLOs (Participant Handbook)

   SLI = good events / valid events

The same source draws a key distinction on how "valid" gets decided depending on the type of system:

"Typically, for data processing systems, validity is determined by input parameters, to scope the SLI to subsets of the data."

process-shipment-manifest is, exactly, a data processing system — not an HTTP API where validity gets decided by hostname or route. This means an event's validity, for this Lambda, gets decided by what kind of input triggered the invocation, not by how the system responded.

Step 1 — Deciding what counts as "valid"

A valid event is one that should count in the SLI's denominator — the complete universe over which "good" versus "bad" gets measured. An invalid event doesn't count for or against: it simply gets excluded from the measurement entirely, because it doesn't represent the kind of real work the SLI is trying to capture.

process-shipment-manifest gets triggered two different ways, and only one of them should count:

Invocation source	Counts as a valid event?	Justification
A real S3 trigger, fired by an actual Andes Cargo customer uploading a manifest	Yes	Exactly the kind of work the SLI should measure — the cited "input parameters" signal applied literally: the event's source (a real upload, not an internal test) is the input parameter that decides validity
Manual test invocation (`aws lambda invoke` from the console or CLI, with a synthetic payload used only to verify the deployed Lambda works)	No	Doesn't represent real traffic from any customer; including it would contaminate the SLI with events nobody outside the engineering team ever saw

This decision has a direct consequence for the dataset lesson 4 is going to use: every row of the 30-day dataset counts only invocations triggered by real manifest uploads to andes-cargo-shipment-docs — never test invocations, no matter how many of those there were on that day.

Step 2 — Deciding what counts as "good"

With the universe of valid events already defined, the next question is harder: of those real invocations, which ones count as success? The source gives two paths: classify an event as good (it enters the numerator) or classify an event as invalid (it leaves the denominator). This lesson picks the first path for the three risks Module 1 already identified — on purpose, so none of those risks stays hidden outside the measurement.

The invocation's real outcome	Counts as a good event?	Why (with the matching Module 1 risk)
Processes the manifest with no exception, within `Timeout: 10`, and writes the correct record to `Shipments`	Yes	The expected case: latency within the limit, no explicit or implicit error
Ends in an unhandled exception	No	Explicit error — lesson 2's errors signal, in its most direct form
Exceeds `Timeout: 10` and gets force-killed	No	The latency risk lesson 4 of Module 1 already identified as a hard limit with no real-duration metric yet
Gets throttled for lack of concurrency (the sixth simultaneous invocation, with all 5 slots occupied)	No	Module 1's saturation risk — and this table's most important decision, see below
Exhausts its two automatic retries and gets discarded without ever being processed	No	Module 1's retries-without-DLQ risk — a manifest that was never processed, with nobody finding out

The decision worth explaining most carefully is the third row. A throttled invocation, technically, didn't even run — Lambda rejected it before executing a single line of the handler. It would be defensible, following the source quoted above, to classify it as invalid instead of bad — excluding it from the denominator instead of counting it as a bad event in the numerator. This guide explicitly chooses not to do that: a throttled invocation represents a real customer's manifest that Andes Cargo failed to process when it should have — exactly the kind of failure Module 1's risk inventory named and left pending to measure. Excluding it from the denominator would make it invisible to the SLI — the error budget would never find out it happened — which would, in practice, be hiding the risk behind a convenient accounting decision. Counting it as a bad event, within valid events, is the only way for this guide's SLI to measure what actually matters to a customer: whether their manifest got processed or not, regardless of the internal technical reason.

The complete SLI definition, in one sentence

With the two previous decisions made, process-shipment-manifest's availability SLI reads, formally, like this:

   process-shipment-manifest availability SLI =

   (invocations triggered by real manifest uploads that finished with no
    exception, within the 10-second timeout, with a correct record
    written to Shipments)
   ------------------------------------------------------------------------
   (all invocations triggered by real manifest uploads, including those
    throttled for saturation and those discarded after exhausting
    retries, excluding only manual test invocations)

This is, literally, the definition lesson 4's dataset implements with two number columns: valid_events (this formula's denominator) and good_events (the numerator). Every row of the 30-day dataset you're going to use in the next lesson is a direct application of this definition, day by day.

Common mistakes

Excluding any uncomfortable outcome from the denominator, instead of counting it as bad ("cleaning up" the SLI). What happens: someone, uncomfortable that throttled invocations lower the SLI, proposes classifying them as "invalid" so they don't count at all. How to spot it: if your justification for excluding a type of event from the denominator is "the number looks better this way" instead of "this event doesn't represent the work the SLI is trying to measure." How to fix it: the source cited in this lesson is explicit that excluding from the denominator and counting as bad are two legitimate mechanisms, but the choice between them should be based on whether the event represents real work, not on its effect on the final number. A throttled invocation represents a real manifest that didn't get processed — it belongs in the denominator, even if that makes the SLI look worse.

Defining "good" only in terms of explicit errors, forgetting implicit ones (from lesson 2). What happens: someone defines "good" as "no exception and no timeout," with no mention at all of whether the record written to Shipments is correct. How to spot it: if your definition of a good event includes no check on the content of the written record, only on whether the invocation "finished without failing." How to fix it: lesson 2 already warned about implicit errors — an invocation that "works" but writes corrupted data. This lesson's definition explicitly includes "with a correct record written to Shipments" precisely to avoid repeating that mistake.

Confusing "valid events" with "every invocation that ever happened" (not filtering anything). What happens: someone includes manual test invocations in the denominator, reasoning that "they're real Lambda invocations too." How to spot it: if your count of valid events for a day includes tests run by the engineering team itself. How to fix it: the source's quote about data processing systems is exactly about this — validity gets decided by the input parameter (is it a real customer upload, or an internal test?), not by whether the Lambda technically ran. An internal test, no matter how many times it's run, should never move Andes Cargo's availability SLI.

Exercises

Exercise 1 — Classify a new event, not mentioned in this lesson. A real process-shipment-manifest invocation, triggered by a genuine manifest upload, runs with no exception, within the timeout, but the manifest itself has a required field left empty (a customer error when generating the file, not a system error), and the Lambda correctly responds with a "manifest rejected: missing field" status, writing no record to Shipments. Is it a valid event? Is it a good event?

See solution

It's a valid event — it comes from a real customer upload, meeting this lesson's validity criterion exactly (the input parameter is a genuine upload, not an internal test). Whether it's good depends on a distinction this lesson didn't explicitly resolve and that's worth pinning down now: the system behaved correctly — it detected a malformed manifest and rejected it with a clear response, instead of failing silently or writing corrupted data. This is, in practice, the system working as expected in the face of invalid customer input, not a system failure. The right classification is to count it as a good event: the availability SLI measures whether process-shipment-manifest did its job (process or correctly reject), not whether each individual manifest arrived well-formed — that would be the customer's responsibility, not the system's. This is exactly the kind of judgment call this lesson insists on making case by case, with explicit justification, instead of automatically assuming "anything other than the full happy path" counts as bad.

Exercise 2 — Explain, in your own words, why "valid events" isn't the same as "total events." A teammate, familiar with the simpler "good events ÷ total events" formula other sources use, asks why this guide insists on the word "valid" instead of "total."

See solution

"Total events" would imply counting absolutely everything that happened, with no filter at all — including internal test invocations, which never represented real work for any customer. "Valid events" deliberately introduces a judgment step before counting: deciding what universe of events represents the work the SLI is trying to measure, and explicitly excluding what doesn't belong to that universe (per the cited source, by input parameters, in the case of a data processing system like this one). The practical difference: if Andes Cargo ran, say, 50 manual test invocations on a particular day and all of them failed (because test payloads are intentionally malformed to verify error handling), a "total events" SLI would count those 50 failures against that day's error budget, completely distorting the measurement of what actually happened to real customers that day. "Valid events" exists precisely so that kind of noise never enters the equation.

Exercise 3 — Defend, against a real objection, the decision to count throttled invocations as "bad" instead of excluding them. An Andes Cargo engineer argues: "throttling an invocation is the system working exactly as designed — protecting concurrency — so it should be invisible to the SLI, not count against it." What do you tell them?

See solution

A complete answer: "The system having worked exactly as designed doesn't mean the outcome for the customer was good — those are two different questions. ReservedConcurrentExecutions: 5 was correctly designed to protect both the function and an external carrier API from a traffic spike (Module 1, lesson 4); that's a defensible design decision. But from the perspective of the customer whose manifest arrived as the sixth simultaneous request, their manifest didn't get processed — period — no matter how good the internal technical reason was. The SLI measures the customer's experience, not the correctness of the system's internal design. If we excluded throttled invocations from the SLI because 'the system did what it was supposed to,' we'd be using the SLI formula to justify the design itself, instead of using it to honestly measure how many real manifests got processed successfully — exactly the trap this lesson deliberately avoids."

Summary and next step

In this lesson you defined, in writing and with explicit judgment, this guide's first real SLI: valid events are invocations triggered by real manifest uploads (never internal tests); good events are the ones that finish with no exception, within the timeout, with a correct record in Shipments — explicitly counting as bad the invocations throttled for saturation and those discarded after exhausting retries, the two risks Module 1 left identified and unresolved. This definition, not an algorithm, is the piece that makes the next lesson's calculator measure something real.

Before moving on you should be able to: recite the complete definition of "valid" and "good" for process-shipment-manifest without looking at this lesson; explain why throttled invocations count as bad and not as invalid; and defend that decision against the objection that "the system worked as designed."

Lesson 4 turns this definition into real code: scripts/error_budget_calculator.py, run for the first time over a fixed 30-day dataset that applies, day by day, exactly the criteria you just defined.

Resources

Google — The Art of SLOs (Participant Handbook) — the exact source for "the proportion of valid events that were good" and how validity gets decided in data processing systems.
Google SRE Workbook — Implementing SLOs — the complementary SLI formulation as an event ratio, with numeric examples.
This same repository, Module 1, lesson 4 (04-hands-on-reading-andes-cargo-like-an-sre.md) — the three risks (saturation, timeout, retries without DLQ) this lesson deliberately folds into the definition of "bad."

← Previous2. Choosing a good SLI: the four golden signals Next →4. Hands-on: the error budget calculator