Module 3: Observability As Sli Input

7. From raw telemetry to a measured SLI

Next →

Description

This is the lesson that closes this whole module's thread. SLO.md (Module 2, lesson 8) defined the SLI, chose the SLO, and measured the error budget over a fixed 30-day dataset — with an explicit warning in its own "Consequences" section: "not a live number [...] but the evidence this SLO was chosen with." Lessons 3 through 6 of this module built, pillar by pillar, a real telemetry pipeline over the same fixed 20-invocation batch. This lesson feeds error_budget_calculator.py — the same compute_sli() function, with no code change at all — that pipeline's numbers, instead of the hand-committed dataset. The calculator runs for real, twice, with two different questions.

Connection to the module

Lesson 6 ended with a number confirmed, three times, through three different paths (representative CloudWatch in lesson 3, real logs in lesson 4, real Prometheus in lesson 6): 20 invocations, 3 errors. This lesson is the first time that number enters the same math function SLO.md already used — the exact point where "observability" and "SRE" stop being two separate words.

Step 1 — Run 1: this module's batch, alone, with no context

The most direct question: if you give compute_sli() exactly the two numbers Prometheus confirmed in lesson 6 — total_valid=20, total_good=17 — treated as if they were the full month, what does the calculator say?

# --- Run 1: ONLY this module's telemetry, as if it were the full month ---
M3_TELEMETRY_BATCH = [(31, 20, 17)]  # (day, valid_events, good_events)

report_1 = budget_report(M3_TELEMETRY_BATCH)
print_report(report_1, "process-shipment-manifest -- M3 telemetry batch only")

What to expect (literal — budget_report() and print_report() are the exact same functions from scripts/error_budget_calculator.py, Module 2, lesson 4, with no changes):

--- process-shipment-manifest -- M3 telemetry batch only ---
Valid events:            20
Good events:             17
Measured SLI:            85.0000%
Target SLO:              99.9%
Budget (monthly):        43.2 minutes
Consumed this period:    6480.00 minutes (15000.0% of budget)
Remaining budget:        -6436.80 minutes

An 85% SLI, a budget consumed at 15,000% — fifteen thousand percent. Before reacting to that number, the next section explains exactly why it's the wrong reading of this data, and what reading is correct.

Why 85% is NOT Andes Cargo's monthly SLI, and why the calculator is right anyway

compute_sli() made no mistake at all — it did exactly what it was asked, with the numbers it was given. The mistake, if there is one, is in the question, not the answer. Two concrete reasons why this result doesn't replace the 99.9098% SLO.md already cited:

Sample size. SLO.md measures an SLI over 12,195 valid events, spread across 30 real days. This run measures an SLI over 20 events, in a minutes-wide window. A proportion of 3 errors out of 20 events is statistically very different from a proportion of 3 errors out of 12,195 — the same absolute error, over a radically smaller denominator, produces a percentage that says nothing reliable about the system's typical behavior.

The batch's design. Lesson 3's batch deliberately has a 15% rate of malformed events — three errors injected on purpose, at fixed positions, to verify the observability pipeline detects them. No real day of Andes Cargo traffic should ever look like this; it's, by design, a stress test of the pipeline, not a sample of customer traffic. Treating this batch as "a normal month" would be, literally, the same mistake lesson 3 already warned about in its common mistakes section.

The right question, then, isn't "does this 85% replace SLO.md's 99.9098%?" — it doesn't. It's: what would happen to the real SLI, measured over Module 2's complete dataset, if this same day of real telemetry were added as one more day?

Step 2 — Run 2: Module 2's real dataset, extended with a real day from this module

from scripts.error_budget_calculator import TRAFFIC_30_DAYS, budget_report, print_report

# --- Run 2: the real 30-day dataset (M2.4) + this module's real data point, as day 31 ---
combined_dataset = TRAFFIC_30_DAYS + M3_TELEMETRY_BATCH

report_2 = budget_report(combined_dataset)
print_report(report_2, "process-shipment-manifest -- 31 days (30 dataset + real M3 telemetry)")

What to expect (literal):

--- process-shipment-manifest -- 31 days (30 dataset + real M3 telemetry) ---
Valid events:            12215
Good events:             12201
Measured SLI:            99.8854%
Target SLO:              99.9%
Budget (monthly):        43.2 minutes
Consumed this period:    49.51 minutes (114.6% of budget)
Remaining budget:        -6.31 minutes

This is the number that actually matters. Module 2's dataset already left only 4.23 minutes of budget unused — 90.2% already consumed, with only four days left to close the month. Adding just one more day, with the same error proportion this module's pipeline really confirmed (3 of 20, 15%), doesn't just consume those remaining 4.23 minutes: it exhausts them and leaves a 6.31-minute deficit. The monthly SLI drops from 99.9098% to 99.8854% — for the first time in this guide, below the 99.9% SLO SLO.md set.

What this run is, and what it isn't

It's important to be precise here, with the same discipline SLO.md already demanded of itself: this run doesn't modify SLO.md. The document keeps citing, as its reference measurement, Module 2's original 30-day dataset — that figure doesn't change retroactively because this module added an illustrative day. What this run demonstrates is something different, and more important for the rest of this guide: the same function, with no code change at all, responds correctly when this module's real telemetry gets integrated correctly — as one more day of an already-existing dataset, not as an isolated replacement for the whole month. That distinction — integrating real telemetry in the right place in the formula, instead of using it to substitute for context it doesn't have — is, in itself, this lesson's technical lesson.

Module 4 is going to build on exactly this result: a burn rate alert that fires when the budget is being consumed at a speed like the one this illustrative day showed — three errors in a handful of minutes is, sustained, a consumption rate far above what a full month allows.

Common mistakes

Citing Run 1's 85% as "Andes Cargo's SLI" out of context (the mistake this lesson exists to prevent). What happens: someone, after this lesson, repeats the number 85% in a conversation — for example, in lesson 8's project — with no clarification that it comes from a 20-invocation batch deliberately designed with a high error rate. How to spot it: if you mention "85%" without saying, in the same sentence, "a small verification batch, not a month of real traffic." How to fix it: Andes Cargo's reference SLI is still the one SLO.md already set — 99.9098% over the original dataset — and the number that really matters from this lesson is Run 2's: 99.8854%, the result of correctly integrating one real day of telemetry into the complete dataset's context.

Thinking Run 2 automatically updates SLO.md (confusing an illustrative exercise with a document change). What happens: someone concludes that, after this lesson, SLO.md should be rewritten with the new 99.8854% SLI. How to spot it: if your plan is to edit SLO.md immediately after this lesson. How to fix it: this run is deliberately illustrative — it shows what would happen to the SLI if this specific day of real telemetry were added to the dataset — not a continuous update process for SLO.md. That document deliberately cites a fixed reference measurement taken in Module 2; Module 4 is what builds the alerting machinery that reacts to data that does change over time, with no need to rewrite SLO.md every time.

Assuming compute_sli() needs code changes to accept real data (underestimating Module 2's calculator design). What happens: someone, seeing this lesson uses data from an observability pipeline instead of a committed dataset, expects compute_sli() itself to need modification. How to spot it: if your plan is "edit error_budget_calculator.py so it accepts Prometheus data." How to fix it: Module 2 lesson 4's exercise 3 already explained why budget_report() receives slo and window_minutes as parameters, not fixed constants — the same design principle applies here: compute_sli() only needs a list of (day, valid, good) tuples, regardless of whether those numbers came from a hand-written dataset or a real PromQL query. No code changes between Module 2 and this lesson — only where the numbers passed to it come from changes.

Exercises

Exercise 1 — Calculate, without running code, how many minutes of budget day 31 alone would consume (not the complete 31-day dataset, just that single day), using error_budget_minutes()'s same logic applied to a 24-hour window instead of 30 days.

See solution

If we treated day 31 as an independent 24-hour window (1,440 minutes), with an observed error rate of 15% (3 of 20): minutes_consumed = 0.15 × 1440 = 216 minutes. Compared against a hypothetical daily budget of (1 - 0.999) × 1440 = 1.44 minutes, that single day, if sustained, would consume 150 times the full daily budget — an even more alarming number than Run 1's 15,000%, precisely because a daily window makes any short burst of errors look proportionally enormous. This calculation confirms, from another angle, the same lesson: the time window matters as much as the error proportion when reading any budget_report() result.

Exercise 2 — Explain, in your own words, why Run 2 (99.8854%) is a more reliable number than Run 1 (85%), even though both use 100% real data from this module. This isn't a question about which data is "more real" — both are.

See solution

Reliability doesn't depend on whether the input data is real — both runs use the same total_valid=20, total_good=17, genuinely measured by Prometheus in lesson 6. It depends on whether the statistical context that data gets interpreted in is the right one. Run 1 treats 20 events as if they represented a full month of the system's typical behavior — a generalization no sample that small can support, and one that also ignores the batch was deliberately designed with a high error rate. Run 2 integrates those same 20 real events as what they genuinely are — one more day, within an already-existing dataset of 12,195 events — letting that day's relative weight within the total be proportional to its real size. It's the difference between "real data, poorly contextualized" and "the same real data, correctly contextualized" — both runs are honest about the data's origin, but only one is honest about what that data can, and can't, say on its own.

Exercise 3 — Predict, without running code, what would happen to the combined SLI if this module's batch had had 20 invocations, all successful (0 errors), instead of 3 failed. Would the resulting SLI be higher, lower, or the same as SLO.md's original 99.9098%? Why?

See solution

The resulting SLI would be slightly higher than the original 99.9098%. With total_valid = 12195 + 20 = 12215 and total_good = 12184 + 20 = 12204 (the original 12,184 good ones plus the 20 new ones, all good): SLI = 12204 / 12215 ≈ 99.9099% — a fraction of a percentage point above the original, because adding events with a perfect success rate (100%), even a small group, always nudges the aggregate average slightly upward, never downward. This exercise confirms, with this lesson's opposite case, the same mechanism: the aggregate SLI moves in the direction of the new data's success rate, with a magnitude proportional to that new data's relative size against the already-existing total — 20 events out of 12,195 move the needle, but only a little, in either direction.

Summary and next step

This lesson closed this module's central thread: it fed error_budget_calculator.py — with no code change at all — the numbers lessons 3 through 6's real observability pipeline produced. The first run, with the 20-invocation batch treated in isolation, gave an alarming result (85% SLI, budget consumed at 15,000%) that turned out to be the wrong reading, not a calculator error — the batch is a stress test of the pipeline, not a representative sample of a month. The second run, integrating that same real day as one more day of Module 2's complete dataset, gave the result that actually matters: the monthly SLI drops to 99.8854%, for the first time below the 99.9% SLO — a real 6.31-minute deficit, not a hypothetical exercise.

Before moving on you should be able to: explain why Run 1's 85% doesn't replace SLO.md's 99.9098%; recite Run 2's exact result (99.8854%, a 6.31-minute deficit); and explain why no code in error_budget_calculator.py changed between Module 2 and this lesson.

Lesson 8, this module's project, documents the complete pipeline — where each piece of data comes from, which instrument produces it, which query reads it — as Andes Cargo's first operational observability runbook, and saves lesson 6's Grafana panel as a reproducible artifact.

Resources

  1. This same repository, Module 2, lesson 4 (04-hands-on-the-error-budget-calculator.md) — error_budget_calculator.py, with no code change in this lesson.
  2. This same repository, Module 2, lesson 8 (08-project-andes-cargos-slo-md.md) — SLO.md, the "Consequences" section this lesson closes.
  3. This same repository, Module 3, lesson 6 (06-hands-on-prometheus-and-grafana-the-stack-the-market-asks-for.md) — the real source of total_valid=20, total_good=17 this lesson uses.
  4. Google SRE Workbook — Implementing SLOs — the error budget formula both of this lesson's runs apply, unchanged, over data from a different origin.
← Previous6. Hands-on: Prometheus and Grafana, the stack the market asks forNext →8. Project: Andes Cargo's observability-to-SLI pipeline