Step 4: Escalation with HITL

Capsule description

Critical alerts require human acknowledgement (ack). If nobody acks within 15 minutes, escalate to the manager. This requires 2 additional workflows: [ALERTS-ACK] (receives acks) and [ALERTS-ESCALATE] (checks pending ones).

---

Workflow [ALERTS-ACK]

Trigger

Webhook that receives the oncall's ack (from the Slack button or email link).

Webhook:
- Method: POST
- Path: /alerts-ack
- Auth: Header X-Ack-Secret

Processing

[Webhook]
   │
[Set: extract alert_id from the payload]
   │
[Sheet: update 'alerts_pending_ack']
- Find row with alert_id
- Set ack_at = $now
- Set ack_by = $json.body.user (if present)
   │
[Slack: confirm to oncall: "Alert {alert_id} acknowledged"]

Link in the original notification

In the original Slack message to the #alerts-critical channel (capsule 05), include a Block Kit button:

{
  "type": "button",
  "text": { "type": "plain_text", "text": "✅ Acknowledge" },
  "url": "https://your-n8n/webhook/alerts-ack?alert_id={{ alert_id }}"
}

Click → GET with alert_id → workflow [ALERTS-ACK] receives it → marks it as acknowledged.

For Slack interactive it really requires a webhook POST with a payload, not a GET. Simplified implementation here; in production use Slack interactivity with a Request URL.

---

Workflow [ALERTS-ESCALATE]

Trigger

Schedule every 5 minutes
Cron: */5 * * * *
TZ: America/Mexico_City

Processing

[Schedule]
   │
[Sheet: read 'alerts_pending_ack']
- Filter: ack_at is empty (no ack) AND created_at > 15 min ago
   │
[IF: any alerts with no ack > 15 min?]
└─ TRUE → escalate

Escalation

[SplitInBatches batch=1]
   │
[Slack: to #manager-alerts]
"🚨 ALERT WITHOUT ACK - Manager attention required
Alert ID: {alert_id}
Metric: {metric}
Time without ack: 15+ minutes
Details: ..."
   │
[Sheet: update alerts_pending_ack]
- Set escalated_at = $now
   │
   └─→ loop back

---

Ensure you don't escalate twice

If escalation runs every 5 min, it could re-escalate the same alert. A filter to avoid this:

[Sheet: filter 'alerts_pending_ack']
- ack_at is empty
- AND created_at > 15 min ago
- AND escalated_at is empty  ← key

Only escalate the ones that haven't been escalated yet.

---

Workflow [ALERTS-MAIN] updated

In the "critical" branch (capsule 05), after notifying:

[Slack #alerts-critical, Email, etc. in parallel]
   │
[Sheet append 'alerts_pending_ack']
- alert_id: {{ $now.toMillis() }} + random
- metric
- severity: 'critical'
- created_at: $now
- ack_at: empty
- escalated_at: empty

Each critical is tracked for possible escalation.

---

Lifecycle of a critical alert

T+0:     Alert detected
         → Slack notification
         → Email to oncall
         → Sheet 'alerts_pending_ack' append
T+15:    [ALERTS-ESCALATE] checks
         → If ack_at empty → escalate to manager
         → Sheet update escalated_at
T+30:    Manager should respond
         (next escalation if you configure it: to director, etc.)

---

Workflow for "alert resolved"

Optional but useful: workflow [ALERTS-RESOLVED] to close alerts when the problem is resolved.

[Webhook /alerts-resolved]
   │
[Sheet update: resolved_at = $now]
   │
[Slack: "Alert X resolved"]

---

Capsule test

Test 1: Critical with fast ack

• Trigger a critical alert
• Verify Slack arrives
• Do a POST to the ack webhook immediately
• Verify the 'alerts_pending_ack' Sheet has ack_at
• 15 min later: escalation should NOT fire

Test 2: Critical with no ack

• Trigger a critical alert
• Do NOT ack
• Wait 15-20 min
• [ALERTS-ESCALATE] should:
  • Detect the alert with no ack
  • Send Slack to the manager
  • Mark escalated_at

Test 3: Multiple criticals

• Generate 3 criticals
• Ack 1, leave 2 without ack
• Escalation should escalate the 2 without ack

---

Validation

• [ALERTS-ACK] receives the webhook and updates the Sheet
• [ALERTS-ESCALATE] runs every 5 min and escalates correctly
• Doesn't escalate twice (escalated_at filter)
• Main Slack message has a functional ack button
• Complete audit trail in the Sheet

What's next: Step 5 — complete system resilience.

---

Created: May 11, 2026 Version: 1.0