Module 4: Alerting On Error Budget Burn Rate
5. Hands-on: a real CloudWatch alarm on the Lambda
Description
Prometheus/Alertmanager (lesson 4) is the engine a team would pick if it already runs a multi-cloud observability stack. But process-shipment-manifest runs on AWS, and AWS already natively publishes exactly the two metrics this module needs (AWS/Lambda/Invocations, AWS/Lambda/Errors) with no exporter of its own, no scrape, no extra container. This lesson builds this module's third engine: aws_cloudwatch_metric_alarm, declared in observability.tf, calculating the same error proportion this module has already evaluated twice — with real CloudWatch metric math, not a precalculated number.
Connection to the module
This lesson adds observability.tf to andes-cargo-infra/, a sibling of finops.tf (which already declares its own alarm, over AWS/Billing, untouched and unrenamed) — the same file-coexistence pattern this ecosystem has used since finops-and-cost-guardrails-guide. terraform validate and terraform plan really ran against this HCL. terraform apply and awslocal cloudwatch describe-alarms stay representative, for the same circumstantial reason as the rest of this ecosystem: with no LOCALSTACK_AUTH_TOKEN exported in this writing environment, the LocalStack container doesn't start.
Why this alarm is simpler than lesson 4's rule — and why that matters
Before the HCL, a design honesty: this lesson's alarm implements a single window, not two. aws_cloudwatch_metric_alarm evaluates a fixed period against a fixed threshold — it has no native mechanism to demand two windows of different duration cross the threshold at once, the way and ignoring(window) does in PromQL. This limitation is real, not an oversight in this lesson, and lesson 6 picks it back up directly when contrasting this alarm with what AWS already automates natively.
What this alarm can do, and does, is something neither lesson 3 nor lesson 4 did yet: calculate burn rate — the error proportion — directly with CloudWatch metric math, with no dependency on a number already calculated externally. It's Table 5-8's Ticket row (threshold 1x, equivalent to a 0.1% error rate), in its single-window version: if the ratio of errors to invocations, measured over a 1-hour window, crosses the 0.1% SLO.md's SLO allows, the alarm fires.
Step 1 — aws_cloudwatch_metric_alarm with metric math, in observability.tf
resource "aws_sns_topic" "reliability_alerts" {
name = "andes-cargo-reliability-alerts"
tags = local.common_tags
}
resource "aws_cloudwatch_metric_alarm" "manifest_error_budget_burn_rate" {
alarm_name = "andes-cargo-manifest-error-budget-burn-rate"
alarm_description = "Fires when process-shipment-manifest's observed error rate crosses the Ticket-tier burn rate (1x, SLO.md's allowed rate of 0.1%) over a 1-hour period."
comparison_operator = "GreaterThanOrEqualToThreshold"
evaluation_periods = 1
threshold = 0.001
treat_missing_data = "notBreaching"
metric_query {
id = "error_ratio"
expression = "errors / invocations"
label = "process-shipment-manifest error ratio"
return_data = true
}
metric_query {
id = "errors"
metric {
metric_name = "Errors"
namespace = "AWS/Lambda"
period = 3600
stat = "Sum"
dimensions = {
FunctionName = "process-shipment-manifest"
}
}
}
metric_query {
id = "invocations"
metric {
metric_name = "Invocations"
namespace = "AWS/Lambda"
period = 3600
stat = "Sum"
dimensions = {
FunctionName = "process-shipment-manifest"
}
}
}
alarm_actions = [aws_sns_topic.reliability_alerts.arn]
ok_actions = [aws_sns_topic.reliability_alerts.arn]
tags = local.common_tags
}
Piece by piece:
- Three
metric_queryblocks, not one.errorsandinvocationsare the two sourcemetric_querys (return_data = false, implicit by default): each declares a realAWS/Lambdametric, with theFunctionName = "process-shipment-manifest"dimension identifying them as belonging to this specific Lambda, exactly like Module 3 lesson 3'sawslocal cloudwatch get-metric-statistics. The thirdmetric_query(error_ratio) is a math expression over the other two —errors / invocations— withreturn_data = true, the signal that this is the value the alarm itself evaluates againstthreshold, not the two source ones. threshold = 0.001. Not a hand-picked number — it's literally1 - SLOfromSLO.md(99.9% monthly → 0.1% allowed error rate), the sameALLOWED_ERROR_RATEModule 2'sburn_rate_of()uses as its denominator. Crossing this threshold over a 1-hour window is, by definition, a burn rate of at least 1x over that window — Table 5-8'sTicketrow, in its single-window version.period = 3600on both source metrics. One hour, soerrors/invocationscount over the same window the threshold is calibrated to represent.treat_missing_data = "notBreaching". If there's no invocation at all in the window (invocations = 0), theerrors / invocationsexpression would be a division by zero, undefined. This line tells CloudWatch to treat that situation as "no data to evaluate, not an alarm," instead of letting the absence of traffic trigger an alarm by error — an explicit design decision, not an unconsidered default.alarm_actionsandok_actions, both pointing at the sameaws_sns_topic. The alarm notifies both when it entersALARMstate and when it returns toOK— lesson 7 of this module routes this same topic to a real channel.
Step 2 — terraform validate and terraform plan: run, literal
terraform initWhat to expect (literal, run to write this lesson):
Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 6.0"...
- Installing hashicorp/aws v6.60.0...
- Installed hashicorp/aws v6.60.0 (signed by HashiCorp)
Terraform has been successfully initialized!
terraform validateWhat to expect (literal):
Success! The configuration is valid.
terraform planWhat to expect (literal — relevant excerpt; Plan: 2 to add because it includes both the alarm and the SNS topic, both new in this file):
# aws_cloudwatch_metric_alarm.manifest_error_budget_burn_rate will be created
+ resource "aws_cloudwatch_metric_alarm" "manifest_error_budget_burn_rate" {
+ actions_enabled = true
+ alarm_actions = (known after apply)
+ alarm_description = "Fires when process-shipment-manifest's observed error rate crosses the Ticket-tier burn rate (1x, SLO.md's allowed rate of 0.1%) over a 1-hour period."
+ alarm_name = "andes-cargo-manifest-error-budget-burn-rate"
+ arn = (known after apply)
+ comparison_operator = "GreaterThanOrEqualToThreshold"
+ evaluation_periods = 1
+ ok_actions = (known after apply)
+ threshold = 0.001
+ treat_missing_data = "notBreaching"
+ metric_query {
+ id = "errors"
+ return_data = false
+ metric {
+ dimensions = {
+ "FunctionName" = "process-shipment-manifest"
}
+ metric_name = "Errors"
+ namespace = "AWS/Lambda"
+ period = 3600
+ stat = "Sum"
}
}
+ metric_query {
+ id = "invocations"
+ return_data = false
+ metric {
+ dimensions = {
+ "FunctionName" = "process-shipment-manifest"
}
+ metric_name = "Invocations"
+ namespace = "AWS/Lambda"
+ period = 3600
+ stat = "Sum"
}
}
+ metric_query {
+ expression = "errors / invocations"
+ id = "error_ratio"
+ label = "process-shipment-manifest error ratio"
+ return_data = true
}
}
# aws_sns_topic.reliability_alerts will be created
+ resource "aws_sns_topic" "reliability_alerts" {
+ arn = (known after apply)
+ id = (known after apply)
+ name = "andes-cargo-reliability-alerts"
}
Plan: 2 to add, 0 to change, 0 to destroy.
Terraform accepted all three metric_query blocks with no schema error, and automatically built the dependency between manifest_error_budget_burn_rate and reliability_alerts from the aws_sns_topic.reliability_alerts.arn reference inside alarm_actions — the same dependency-graph mechanism you already saw in terraform-and-iac-guide and in the direct precedent from finops-and-cost-guardrails-guide, Module 5, lesson 4.
Step 3 — The apply attempt: really run, same root cause as the rest of this ecosystem
terraform apply -auto-approve \
-target=aws_sns_topic.reliability_alerts \
-target=aws_cloudwatch_metric_alarm.manifest_error_budget_burn_rateWhat to expect (literal, run to write this lesson):
Error: creating SNS Topic (andes-cargo-reliability-alerts): operation error SNS:
CreateTopic, exceeded maximum number of attempts, 9, https response error
StatusCode: 0, RequestID: , request send failed, Post "http://localhost:4566/":
dial tcp [::1]:4566: connect: connection refused
The same connection refused, the same root cause as every apply attempt in this ecosystem since finops-and-cost-guardrails-guide: with no LOCALSTACK_AUTH_TOKEN exported, the LocalStack container never starts in this writing environment. Terraform tries to create aws_sns_topic.reliability_alerts first (which the alarm depends on, via alarm_actions), and fails there — the alarm doesn't even get attempted in this specific run.
The real difference that matters isn't in this error message — it's in what would happen if you resolved layer 1. CloudWatch, including PutMetricAlarm with metric math, is confirmed on LocalStack's free Hobby plan (the same source this guide's Module 3 already verified). With a real LOCALSTACK_AUTH_TOKEN and the container running, this same apply would complete successfully.
Representative verification: awslocal cloudwatch describe-alarms
What to expect (representative — same reason as the rest of this ecosystem: this writing environment has no LOCALSTACK_AUTH_TOKEN exported, so the container never starts to respond to this command; reconstructed field by field from Step 1's real HCL):
awslocal cloudwatch describe-alarms --alarm-names andes-cargo-manifest-error-budget-burn-rate \
--query 'MetricAlarms[0].{Name:AlarmName,State:StateValue,Threshold:Threshold,Comparison:ComparisonOperator}'{
"Name": "andes-cargo-manifest-error-budget-burn-rate",
"State": "INSUFFICIENT_DATA",
"Threshold": 0.001,
"Comparison": "GreaterThanOrEqualToThreshold"
}Name, Threshold, and Comparison are literal, taken directly from Step 1's HCL. State: INSUFFICIENT_DATA is the state CloudWatch assigns to any freshly created alarm, before it accumulates its first complete data point — unlike finops-and-cost-guardrails-guide's billing alarm (which stays in that state forever, because EstimatedCharges never has real data in a $0 lab), this alarm does have a real, available data source: process-shipment-manifest really gets invoked and really generates Invocations/Errors metrics in CloudWatch every time it processes a manifest — the same batch from Module 3, lesson 3. With LocalStack running and real traffic flowing through the Lambda, this alarm would actually reach OK or ALARM, unlike the billing alarm that never leaves INSUFFICIENT_DATA.
Reading the result: the same decision, a third engine
THE SAME DECISION, THREE ENGINES -- CONFIRMED SO FAR
Python (M4.3) Prometheus/Alertmanager (M4.4) CloudWatch (M4.5)
────────────── ─────────────────────────── ─────────────────
3 severities 3 PromQL rules with 1 alarm, 1 window,
(Page fast/slow, Ticket) ignoring(window) real metric math
evaluated in memory evaluated against a (errors/invocations)
real exporter threshold = 0.001
bad_week -> FIRES x3 bad_week -> firing x3 (real validate/plan;
normal -> doesn't fire normal -> never appears representative apply)All three engines implement the same question — did the error budget's consumption cross the threshold SLO.md allows? — with three different levels of sophistication: pure Python (no infrastructure), Prometheus/Alertmanager (real multi-window, portable across providers), CloudWatch (a single window, but native to AWS, with no exporter or extra container to maintain). Lesson 6 names the piece this third engine is missing — and which managed AWS product already solves it natively.
Common mistakes
Writing the metric math expression as invocations / errors instead of errors / invocations (swapping numerator and denominator). What happens: someone, declaring expression = "invocations / errors", gets a number that goes up when the system improves, instead of down — exactly the opposite of what a GreaterThanOrEqualToThreshold threshold needs to make sense. How to spot it: if your alarm would fire on healthy traffic and stay quiet during a real incident. How to fix it: the ratio this lesson needs is "fraction of invocations that failed" — errors / invocations — the same orientation Module 2's compute_sli() uses (though that function calculates the complement, good / valid). Always check which direction makes sense with the chosen comparison_operator: GreaterThanOrEqualToThreshold needs a number that rises when things get worse.
Forgetting treat_missing_data = "notBreaching", and not understanding why the alarm behaves differently in a window with no traffic (a silent default value). What happens: someone omits this line, relying on CloudWatch's default behavior, and in a window with no invocations at all, the alarm ends up in an unexpected state. How to spot it: if your alarm changes state during zero-traffic hours (for example, overnight, if Andes Cargo had a traffic pattern with inactive hours) with no real error having happened. How to fix it: CloudWatch's default for treat_missing_data is "missing", which can leave the alarm in an ambiguous state when there's no data to evaluate — "notBreaching", explicitly declared in this lesson, tells CloudWatch to treat the absence of data as "everything's fine," preventing a traffic-free window (division by zero avoided, not an error condition) from firing an alarm with no real failure behind it.
Assuming this alarm implements the same multi-window pattern as lesson 4, just because it uses the same conceptual threshold (1x, Ticket tier) (confusing "same math" with "same coverage"). What happens: someone, after seeing threshold = 0.001 match Table 5-8's Ticket row, assumes this alarm has the same false-positive protection as the Alertmanager rule. How to spot it: if you expect this alarm, like lesson 4's rule, to automatically stop firing as soon as a one-off problem resolves, without waiting for the full 1-hour window to "clear." How to fix it: this lesson declares it explicitly in its opening section — a single window, with no short-window confirmation. It's a real limitation of aws_cloudwatch_metric_alarm in its simplest form, not an oversight in this guide; lesson 6 names the managed AWS piece that does solve this natively.
Exercises
Exercise 1 — Calculate, without running anything, whether this alarm would fire on Module 3 lesson 3's 20-invocation batch (Invocations: 20.0, Errors: 3.0), assuming those 20 invocations happened within a single 1-hour window.
See solution
Yes, it would fire. error_ratio = errors / invocations = 3 / 20 = 0.15 (15%), far above threshold = 0.001 (0.1%) with comparison_operator = "GreaterThanOrEqualToThreshold". This result makes sense: Module 3's batch was deliberately designed with a high error rate (15%) to verify the observability pipeline detects failures, not to represent normal production traffic — the same warning Module 3, lesson 3 already gave in its "Common mistakes" ("20 invocations [...] aren't a representative sample of the monthly SLI").
Exercise 2 — Explain why this alarm uses metric_query with an expression, instead of creating the alarm directly on the Errors metric with a threshold in an absolute number of errors (for example, "fire if Errors sums to 3 or more in an hour").
See solution
A threshold in an absolute number of errors isn't tied to any traffic volume — 3 errors out of 20 invocations (15%) is a very different situation from 3 errors out of 3,000 invocations (0.1%), but a "3 errors" threshold would fire equally in both cases. The ratio (errors / invocations), by contrast, directly measures the same thing SLO.md defines as the SLI — a rate, not a count — so this alarm's threshold = 0.001 is directly comparable to the error rate the SLO allows (1 - 0.999 = 0.001), regardless of how much traffic process-shipment-manifest receives in a given hour. It's the same principle that led Module 2's compute_sli() to calculate a ratio, not an absolute count.
Exercise 3 — Design, in prose (with no HCL written yet), a second CloudWatch alarm that gets closer to the multi-window pattern, using two alarms combined with aws_cloudwatch_composite_alarm. What would each of the two simple alarms declare, and what expression would the composite alarm combine them with?
See solution
You'd declare two aws_cloudwatch_metric_alarms nearly identical to this lesson's, but with a different period in their source metric_querys — one with period = 3600 (1 hour, the long window) and another with period = 21600 (6 hours, approximating Table 5-8's Ticket row's short window, which in real production would be 6 hours for that row) — each with its own alarm_name. Then, an aws_cloudwatch_composite_alarm with an alarm_rule like "ALARM(long_window_alarm) AND ALARM(short_window_alarm)" — CloudWatch's real syntax for combining simple alarms' states with boolean logic — would reproduce the same two-window AND condition PromQL expresses with and ignoring(window). This lesson doesn't build it — outside its declared scope — but the mechanism exists natively in CloudWatch, and it's exactly the direction a real team would take this alarm if they decided to close the gap with Google SRE's complete pattern without leaving the AWS ecosystem.
Summary and next step
This lesson built this module's third engine: aws_cloudwatch_metric_alarm with real metric math (errors / invocations), over process-shipment-manifest's native AWS/Lambda metrics, with a threshold (0.001) taken directly from SLO.md. terraform validate and terraform plan really ran, with no error, automatically building the dependency toward the new aws_sns_topic.reliability_alerts. The apply attempt also really ran and failed for the same circumstantial reason as always (connection refused, no LOCALSTACK_AUTH_TOKEN) — but, unlike finops-and-cost-guardrails-guide's billing alarm, this alarm would have real data to evaluate on a normally running LocalStack, because process-shipment-manifest really generates traffic. The real limitation, declared honestly from the start: this alarm evaluates a single window, not two — CloudWatch, in its simplest form, doesn't have the same confirmation mechanism as Prometheus/Alertmanager.
Before moving on you should be able to: explain what each of this alarm's three metric_querys does; calculate by hand whether a given invocations/errors batch would fire this alarm; and name this alarm's real limitation against lesson 4's multi-window pattern.
Lesson 6 names the piece this third engine is missing: CloudWatch Application Signals, the managed AWS product that does implement multi-window burn rate natively — contrasted line by line against what this module built by hand.
Resources
- Terraform Registry —
aws_cloudwatch_metric_alarm— the resource's complete schema reference, includingmetric_query. - AWS Docs — Using metric math — the official reference for the metric math expressions used in Step 1.
- LocalStack Docs — CloudWatch — confirmation of the Hobby plan, the source for this lesson's "representative" label.
finops-and-cost-guardrails-guide, Module 5, lesson 4 — the direct precedent for a real CloudWatch alarm in this ecosystem, and the same honesty pattern (realvalidate/plan, representativeapply).- This same repository, Module 2, lesson 8 (
08-project-andes-cargos-slo-md.md) —SLO.md, the exact source for this lesson'sthreshold = 0.001.