Module 4: Alerting On Error Budget Burn Rate
7. Hands-on: routing the alert
Description
Lesson 5's CloudWatch alarm already publishes to aws_sns_topic.reliability_alerts when it crosses the threshold — but an SNS topic with no subscription is an alarm shouting into an empty room. This lesson closes that last stretch: it declares a real SNS subscription (aws_sns_topic_subscription), validates the exact pattern a real team would use to connect that alarm to PagerDuty, and names, with honesty about its cost, the final destination a real production team would actually use — PagerDuty or Opsgenie — without building it, because neither has a free tier equivalent to the rest of this $0 lab.
Connection to the module
This lesson extends observability.tf — lesson 5's same file, with no new SNS topic declared — adding the piece missing between "the alarm fires" and "someone finds out." terraform validate and terraform plan really ran against this new HCL. Lesson 8, this module's project, documents this complete chain — alarm → topic → subscription — as part of Andes Cargo's alerting policy.
Step 1 — The missing link: a real subscription on the already-declared topic
aws_sns_topic.reliability_alerts has already existed since lesson 5. This lesson adds a subscription to it, with two documented routes — one that would really work against a real AWS account with no additional cost, and another that models the exact pattern that would connect this alarm to PagerDuty:
# Route 1: email, $0, would really work against a real AWS account
# (SNS doesn't charge for email notifications up to a high volume -- see Resources).
resource "aws_sns_topic_subscription" "reliability_alerts_email" {
topic_arn = aws_sns_topic.reliability_alerts.arn
protocol = "email"
endpoint = "reliability-team@andescargo.io"
}
# Route 2: the real pattern a production team would use for PagerDuty --
# HTTPS, with the integration URL PagerDuty generates when the service is created
# (see Resources). "REPLACE_WITH_INTEGRATION_KEY" is a deliberate placeholder:
# that key is generated by PagerDuty per account, never a value this guide could fix.
resource "aws_sns_topic_subscription" "reliability_alerts_pagerduty" {
topic_arn = aws_sns_topic.reliability_alerts.arn
protocol = "https"
endpoint = "https://events.pagerduty.com/x-ere/REPLACE_WITH_INTEGRATION_KEY"
}
Two routes, two different honesty levels. The first (email) is a complete, real resource: protocol = "email" doesn't depend on any third-party service, and on a real AWS account with a working LocalStack, this apply would complete successfully and a real subscription-confirmation email would actually arrive. The second (https, pointed at PagerDuty) is the exact pattern PagerDuty's official documentation specifies for its Amazon CloudWatch integration — HTTPS protocol, the URL in the events.pagerduty.com/x-ere/<integration_key> format — but with the integration key as an explicit placeholder: that key is generated by PagerDuty when a service is created inside your own account, never a generic value this guide could declare ahead of time.
Step 2 — terraform validate and terraform plan: run, literal
terraform validateWhat to expect (literal):
Success! The configuration is valid.
terraform planWhat to expect (literal — relevant excerpt; Plan: 4 to add because it also includes lesson 5's alarm and SNS topic, all in observability.tf):
# aws_sns_topic_subscription.reliability_alerts_email will be created
+ resource "aws_sns_topic_subscription" "reliability_alerts_email" {
+ arn = (known after apply)
+ confirmation_timeout_in_minutes = 1
+ confirmation_was_authenticated = (known after apply)
+ endpoint = "reliability-team@andescargo.io"
+ endpoint_auto_confirms = false
+ filter_policy_scope = (known after apply)
+ id = (known after apply)
+ owner_id = (known after apply)
+ pending_confirmation = (known after apply)
+ protocol = "email"
+ raw_message_delivery = false
+ topic_arn = (known after apply)
}
# aws_sns_topic_subscription.reliability_alerts_pagerduty will be created
+ resource "aws_sns_topic_subscription" "reliability_alerts_pagerduty" {
+ arn = (known after apply)
+ confirmation_timeout_in_minutes = 1
+ confirmation_was_authenticated = (known after apply)
+ endpoint = "https://events.pagerduty.com/x-ere/REPLACE_WITH_INTEGRATION_KEY"
+ endpoint_auto_confirms = false
+ filter_policy_scope = (known after apply)
+ id = (known after apply)
+ owner_id = (known after apply)
+ pending_confirmation = (known after apply)
+ protocol = "https"
+ raw_message_delivery = false
+ topic_arn = (known after apply)
}
Plan: 4 to add, 0 to change, 0 to destroy.
raw_message_delivery = false (the resource's default value, not declared explicitly in Step 1) is exactly what PagerDuty's own documentation asks for: "Ensure that the Enable raw message delivery checkbox is unchecked" — with raw_message_delivery = false, SNS wraps every notification in its own JSON envelope (with topic metadata, timestamp, signature), the format PagerDuty's CloudWatch integration expects to parse the alarm correctly. Setting it to true would deliver the "raw" message, with no envelope — breaking parsing on PagerDuty's side.
Step 3 — The apply attempt: representative, same root cause
What to expect (representative — same reason as lesson 5: with no LOCALSTACK_AUTH_TOKEN, the LocalStack container doesn't start in this writing environment):
terraform apply -auto-approve \
-target=aws_sns_topic_subscription.reliability_alerts_emailError: creating SNS Topic (andes-cargo-reliability-alerts): operation error SNS:
CreateTopic, exceeded maximum number of attempts, 9, https response error
StatusCode: 0, RequestID: , request send failed, Post "http://localhost:4566/":
dial tcp [::1]:4566: connect: connection refused
with aws_sns_topic.reliability_alerts,
on observability.tf line 1, in resource "aws_sns_topic" "reliability_alerts":
1: resource "aws_sns_topic" "reliability_alerts" {
The same connection refused from every apply attempt in this module — and, again like lesson 5, Terraform doesn't even get to attempt the subscription itself: since reliability_alerts_email depends on aws_sns_topic.reliability_alerts.arn, Terraform tries to create the topic first, and fails there, before touching the subscription. The real difference, again, isn't in the error — it's in what would happen with a normally running LocalStack: the email subscription would apply for real and completely (SNS is confirmed on LocalStack's Hobby plan, alongside CloudWatch); the https subscription toward PagerDuty would also apply on the AWS/LocalStack side — but the endpoint itself (events.pagerduty.com) is a real external service, outside any simulation, and would only respond with a valid confirmation if the integration key were a real one, generated inside a real PagerDuty account — the exact reason this lesson can't, even with the token exported, complete that second route end to end.
Why PagerDuty and Opsgenie stay named, not built
Both are paid SaaS, with no free tier equivalent to the rest of this $0 lab. Both official documentation sets confirm the same integration pattern — an SNS HTTPS subscription toward an account-specific URL — with one relevant difference between the two: PagerDuty documents a predictable URL format (events.pagerduty.com/x-ere/<key>), while Opsgenie generates a complete integration URL per account, with no fixed public format, only obtainable by first creating a real integration inside its console ("copy the endpoint URL generated for your account").
THE COMPLETE PATTERN, THE SAME ALARM'S THREE POSSIBLE DESTINATIONS
aws_cloudwatch_metric_alarm (M4.5)
|
v
aws_sns_topic.reliability_alerts (M4.5)
|
┌──────┼──────────────────┐
v v v
email PagerDuty Opsgenie
($0, (HTTPS, SaaS, (HTTPS, SaaS,
real) events.pagerduty account-generated
.com/x-ere/<key>) URL)Neither PagerDuty nor Opsgenie gets built in this lesson — they get named, with the exact integration pattern verified against their own official documentation, and with the honesty that a real, budgeted Andes Cargo team would end up on exactly one of the two, not on an email. This lesson's email route is the complete $0 substitute: it really works, at no cost, but with no automatic escalation, no mobile app, no built-in on-call rotation — the same absences this guide's Module 5, about on-call, is going to name with the same honesty.
Common mistakes
Declaring raw_message_delivery = true "because it sounds simpler" (unknowingly breaking the contract PagerDuty/Opsgenie expects). What happens: someone, seeing raw_message_delivery controls whether SNS wraps the message or delivers it as-is, assumes "as-is" is simpler and switches it to true. How to spot it: if your subscription toward PagerDuty or Opsgenie stops generating recognizable incidents, even though the notifications do arrive. How to fix it: both services expect, on their integration side, the complete JSON envelope SNS adds by default (raw_message_delivery = false, as this lesson leaves it) — removing it doesn't simplify anything, it breaks the automatic parsing the receiving side needs to recognize the message comes from a specific CloudWatch alarm.
Inventing a "sample" PagerDuty integration key that looks real, instead of using an explicit placeholder (fabricating data that looks verifiable without being so). What happens: someone, uncomfortable leaving REPLACE_WITH_INTEGRATION_KEY visible, replaces that text with an invented alphanumeric string that looks like a real PagerDuty key. How to spot it: if your HCL has a value in endpoint that looks like a real key but doesn't come from any real PagerDuty account. How to fix it: an explicit, readable placeholder (REPLACE_WITH_INTEGRATION_KEY) is more honest than an invented value that looks real — the same discipline this ecosystem already follows with any sample credential, from access_key = "test" in every LocalStack provider "aws" block to this case.
Assuming declaring the email subscription in Terraform already means the email arrived (confusing declaring with delivering). What happens: someone reads Route 1's HCL and assumes that, as soon as it's applied, the reliability team is already receiving alerts by email, with no additional step. How to spot it: if your understanding of SNS's email flow doesn't include a confirmation step. How to fix it: an email subscription in SNS doesn't go active immediately — SNS first sends a confirmation email to the declared endpoint, and the subscription stays in PendingConfirmation until someone clicks that confirmation link. It's a manual step, outside Terraform's scope, that not even a successful apply completes on its own.
Exercises
Exercise 1 — Explain why this lesson declares two different aws_sns_topic_subscriptions on the same aws_sns_topic, instead of replacing one with the other. What does Andes Cargo gain by having both routes active at once, on a real account?
See solution
An SNS topic supports multiple simultaneous subscriptions, each receiving an independent copy of every published message — they aren't mutually exclusive alternatives, they're parallel channels. Having both active at once gives Andes Cargo real redundancy: if the PagerDuty integration fails silently (an expired integration key, for example), the email keeps arriving as a backup, without the alarm going completely silent. It's the same "don't depend on a single alert engine" principle lesson 1 of this module already introduced when explaining why three different engines exist (Python, Prometheus/Alertmanager, CloudWatch) — here, applied at the notification-channel level instead of the decision-engine level.
Exercise 2 — A teammate proposes using protocol = "sms" instead of "email" for this lesson's $0 route, arguing an SMS arrives faster at 3 AM. Is this route really $0, just like email?
See solution
Not in the same way. Unlike SNS's email notifications (free up to a high monthly volume), SNS's sms notifications do have a per-message cost from the very first SMS, with no equivalent free tier — a small cost, but real, and different from zero. This is exactly the kind of distinction this whole ecosystem, since finops-and-cost-guardrails-guide, demands verifying before assuming any AWS resource is automatically $0: aws_sns_topic_subscription with protocol = "email" is; with protocol = "sms", it isn't. Delivery speed is a real advantage of SMS, but not a free one, and this guide declared $0 as an explicit commitment since Module 1.
Exercise 3 — Design, in prose, how you'd verify — with no money spent, no real PagerDuty account — that this lesson's Route 2 HCL (aws_sns_topic_subscription.reliability_alerts_pagerduty) is syntactically correct and matches PagerDuty's official pattern, with no way to confirm it with a real apply against the external service.
See solution
The verification available with no money spent is exactly what this lesson already did: terraform validate confirms the resource is well-formed per the AWS provider's schema (protocol, topic_arn, endpoint as a valid string) — that doesn't depend at all on whether the destination URL is real or a placeholder. To confirm the URL's format matches what PagerDuty expects, the verification is documentary, not executable: comparing the declared pattern (https://events.pagerduty.com/x-ere/<key>) against PagerDuty's official documentation, exactly as this lesson already did by citing the source in the "Why PagerDuty and Opsgenie stay named" section. The only verification that would genuinely require a real PagerDuty account is the message's actual delivery — that one honestly stays outside this $0 lab's scope.
Summary and next step
This lesson closed the chain lesson 5 left open: aws_cloudwatch_metric_alarm → aws_sns_topic → aws_sns_topic_subscription, with two routes declared — one completely $0 and functional (email), another that models the exact pattern, verified against PagerDuty's official documentation, a real team would use to connect this alarm to a real on-call system. terraform validate and terraform plan really ran, with no errors; the apply attempt failed for the same circumstantial reason as always. PagerDuty and Opsgenie were named, with their real integration pattern cited, never built — both paid SaaS, with no $0 tier in this ecosystem.
Before moving on you should be able to: explain the real cost difference between an email subscription and an sms one in SNS; explain why raw_message_delivery must stay false for a PagerDuty/Opsgenie integration; and name the complete three-resource chain that connects an alarm to a real notification.
Lesson 8 closes this module with the project: a document that gathers Alertmanager's (lesson 4) and CloudWatch's (lesson 5) burn rate rules, tested with a forced drill — this module's same result, "bad week" fires, "normal" doesn't, confirmed once more, with all three complete engines.
Resources
- Terraform Registry —
aws_sns_topic_subscription— the resource's complete schema reference. - PagerDuty — Amazon CloudWatch Integration Guide — the exact source for the URL format and the
raw_message_deliveryconfiguration cited in this lesson. - Opsgenie (Atlassian) — Integrate Opsgenie with Amazon CloudWatch — the same integration pattern, documented on Opsgenie's side.
- AWS Docs — Amazon SNS pricing — the source for exercise 2's distinction between
email(free up to a high volume) andsms(costs from the first message). - This same repository, Module 4, lesson 5 (
05-hands-on-a-real-cloudwatch-alarm-on-the-lambda.md) —aws_sns_topic.reliability_alerts, the resource this lesson extends with a real subscription.