Module 8: Project — Scheduled Alerting System

Step 5: Complete Resilience and Error Handling

Next →

Capsule description

Apply all of M06 (error handling) to the system. Retry strategies, fallbacks, Error Workflow, observability.

Apply retry strategies

On all critical nodes of the 3 workflows:

Sheets reads/writes

  • Retry On Fail: ON
  • Max Tries: 3
  • Wait Between Tries: 2000ms

Slack notifications

  • Retry On Fail: ON
  • Max Tries: 3
  • Wait: 2000ms
  • Continue On Fail: ON (Slack down must not stop everything)

Email send

  • Retry On Fail: ON
  • Max Tries: 3
  • Wait: 5000ms
  • Continue On Fail: ON

Fallbacks

If Slack fails

[Slack with Continue on Fail]
   │
[IF: error?]
└─ TRUE → [Alternative email to oncall]  // fallback to email

If the main channel (Slack) fails, use email as a fallback.

If Email fails

Consider SMS via Twilio as a last-resort fallback for criticals.

Error Workflow

Create [GUARDIAN-ALERTS]

[Error Trigger]
   │
[Set: enrich]
- workflow_name: $json.workflow.name
- timestamp
- severity: critical (because it's an alerting system — ironic if it fails)
   │
[Slack #ops-critical: "Error in the alerting system"]
   │
[Email to CTO/admin: "URGENT: alert system failure"]
   │
[Sheet log 'system_failures']

Assign to the 3 workflows

  • [ALERTS-MAIN] → Error Workflow: [GUARDIAN-ALERTS]
  • [ALERTS-ACK] → same
  • [ALERTS-ESCALATE] → same

Observability

Sheet system_log

Each [ALERTS-MAIN] execution logs:

- timestamp
- duration_seconds
- alerts_generated (count)
- alerts_critical
- alerts_high
- ok_metrics
- errors_encountered (count)
- status: 'success' | 'partial_failure' | 'error'

Manual dashboard

Separate workflow [ALERTS-DASHBOARD] that runs daily:

[Daily Schedule 9am]
   │
[Sheet read: system_log last 7 days]
   │
[Set: compute metrics]
- runs_total
- failure_rate: errors / total
- avg_duration
- pct_runs_with_alerts
   │
[Slack to #monitoring: "Weekly health report of the alerting system"]

"The alarm system is dead" alarm

Famous trap: your alerting system fails. How do you find out? You need a "watcher of watchers".

Implementation

[ALERTS-HEARTBEAT]:

[Schedule every 30 min]
   │
[Sheet read: system_log]
   │
[IF: any execution in the last 30 min?]
├─ TRUE → all OK, log heartbeat
└─ FALSE → [Slack to admin: "ALERTS-MAIN hasn't run in 30+ min — system possibly down"]

Another workflow that verifies the main one is alive.

Configuration validation

At the start of [ALERTS-MAIN], a sanity check:

[Initial check]
- Sheets accessible?
- Slack credential works?
- Email credential works?
   │
[IF: any check fails]
└─ TRUE → notify admin, don't proceed

Without this, the system runs with broken credentials and fails silently.

Idempotency

If the same Schedule fires twice (from an n8n bug or something odd), don't duplicate alerts:

Approach

Each execution generates a unique execution_id. When logging, check if it already exists:

[Sheet check: does an alert already exist with this timestamp + metric?]
├─ TRUE → skip (duplicate)
└─ FALSE → process normally

Resilience test

Test: Slack down

Simulate: change the Slack credential to an invalid one.

  • [ALERTS-MAIN] should:
    • Retry 3 times (each one fails)
    • Continue on Fail
    • If configured, fall back to email
    • The workflow ends with partial success

Test: Sheet down

Simulate: revoke access to the Sheet from Google.

  • [ALERTS-MAIN] retry → fails → Error Workflow fires
  • You get an alert "the alerting system failed"

Test: Workflow doesn't run

Deactivate [ALERTS-MAIN]. After 30 min:

  • [ALERTS-HEARTBEAT] detects it
  • Slack to admin

Validation

  • Retry configured on critical nodes
  • Continue on Fail where appropriate
  • Fallbacks (Slack → email)
  • [GUARDIAN-ALERTS] assigned to the 3 workflows
  • Sheet system_log with observability
  • [ALERTS-HEARTBEAT] running
  • Initial validation implemented

What's next: Final test, deploy and monitoring.

Created: May 11, 2026 Version: 1.0

← PreviousStep 4: Escalation with HITLNext →Final Test, Deploy and Monitoring