Module 6: Rollback And Safety Nets

8. Project: Andes Cargo's safety net

Next →

Description

This is the project that closes Module 6. Lessons 2 through 7 built each piece separately —the rollback mechanism, branch protection, the incident analysis, and the real guardrail—. This project runs Andes Cargo's complete pipeline, with lesson 7's guardrail already permanently integrated into ci.yml, and tests it twice, on two scenarios you already know: a harmless change, that passes end-to-end like any normal Pull Request in this guide; and an attempt to destroy the Shipments table, which the pipeline stops before any chance to apply exists.

Connection to the module

This project doesn't introduce any new mechanism — it's the final confirmation that lesson 7's guardrail isn't an isolated experiment: it lives, permanently, inside the same ci.yml that runs on every Andes Cargo Pull Request since Module 3. With this, all of Module 6 closes, and Module 7 can locate everything built within CI/CD's broader landscape, without building anything new on andes-cargo-infra/.

The complete pipeline, at a glance

  CASE 1 — HARMLESS CHANGE                        CASE 2 — DESTRUCTION ATTEMPT

  Pull Request #42                                "PR" that removes Shipments
  (feature/add-shipment-tags → main)              (seeded state fixture,
       │                                            HCL without the table)
       ▼                                                 │
  ci.yml (pull_request)                                  ▼
   fmt → init → validate → plan                   guardrail-demo.yml
   Plan: 12 to add, 0 destroy                       (workflow_dispatch)
       │                                            terraform state push (fixture)
       ▼                                             → plan -refresh=false
  Guardrail — block any plan                        Plan: 10 to add, 1 to destroy
  that destroys Shipments                                 │
   "Guardrail passed"                                     ▼
       │                                            Guardrail — block any plan
       ▼                                            that destroys Shipments
  Publish + upload artifact                          "Guardrail failed"
       │                                            exit 1
       ▼                                                 │
  🏁 Job succeeded                                        ▼
                                                     🏁 Job failed
                                                     (apply.yml NEVER triggers:
                                                      there's no artifact to download)

The detail that makes this diagram the final proof of this module's thesis: in Case 2, the job fails before the Upload the plan for apply.yml to use later step — the same mechanism you already saw in Module 5 with needs: (a failing Stage stops the next one), now applied inside a single job: no step after the guardrail runs if the guardrail fails. apply.yml never has an artifact to download, so it could never apply anything, even if someone triggered it manually.

Step 1 — Confirming the starting point

cd andes-cargo-infra
git log --oneline -5

What to expect (representative for the hashes, literal for the structure — assuming you already completed this module's lessons 3, 4, and 7):

<hash> ci.yml: add a guardrail step that blocks any plan destroying the Shipments table; add guardrail-demo.yml to test it against a seeded fixture
<hash> Add pr-event-revert.json to simulate the revert PR with act
<hash> Revert "Add Compliance tag to the shipment-docs bucket"
<hash> drift.yml: schedule-based drift detection with terraform_wrapper: false
<hash> apply.yml: needs, artifact download, and the real terraform apply attempt

git status

What to expect (literal):

On branch main
nothing to commit, working tree clean

Case 1 — The harmless change: ci.yml runs completely, guardrail included

Run ci.yml, with the guardrail already integrated, on exactly the same PR #42 that has run through this guide since Module 2:

export ARTIFACT_ADDR=$(ipconfig getifaddr en0)   # Linux: hostname -I | awk '{print $1}'
rm -rf .artifacts && mkdir -p .artifacts

act pull_request -e .github/act-events/pr-event.json -j terraform-checks \
  --artifact-server-path ./.artifacts \
  --artifact-server-addr "$ARTIFACT_ADDR"

What to expect (literal output, executed to write this lesson — the complete job, eleven steps, all green):

[ci/terraform-checks] ⭐ Run Set up job
[ci/terraform-checks]   ✅  Success - Set up job
[ci/terraform-checks] ⭐ Run Main Check out andes-cargo-infra
[ci/terraform-checks]   ✅  Success - Main Check out andes-cargo-infra [41.6135ms]
[ci/terraform-checks] ⭐ Run Main Set up Terraform
[ci/terraform-checks]   ✅  Success - Main Set up Terraform [2.248916334s]
[ci/terraform-checks] ⭐ Run Main Terraform format check
[ci/terraform-checks]   ✅  Success - Main Terraform format check [155.826625ms]
[ci/terraform-checks] ⭐ Run Main Terraform init
[ci/terraform-checks]   ✅  Success - Main Terraform init [15.009926792s]
[ci/terraform-checks] ⭐ Run Main Terraform validate
[ci/terraform-checks]   ✅  Success - Main Terraform validate [1.903371334s]
[ci/terraform-checks] ⭐ Run Main Install awslocal
[ci/terraform-checks]   ✅  Success - Main Install awslocal [11.256066667s]
[ci/terraform-checks] ⭐ Run Main Confirm the runner can reach LocalStack on the host
[ci/terraform-checks]   | Could not connect to the endpoint URL: "http://host.docker.internal:4566/"
[ci/terraform-checks] Failed but continue next step
[ci/terraform-checks]   ❌  Failure - Main Confirm the runner can reach LocalStack on the host [9.37373375s]
[ci/terraform-checks] ⭐ Run Main Install tflocal
[ci/terraform-checks]   ✅  Success - Main Install tflocal [2.799894084s]
[ci/terraform-checks] ⭐ Run Main Terraform plan
[ci/terraform-checks]   | Plan: 12 to add, 0 to change, 0 to destroy.
[ci/terraform-checks]   ✅  Success - Main Terraform plan [5.791599917s]
[ci/terraform-checks] ⭐ Run Main Guardrail — block any plan that destroys the Shipments table
[ci/terraform-checks]   | Guardrail passed: no destroy action found for aws_dynamodb_table.shipments.
[ci/terraform-checks]   ✅  Success - Main Guardrail — block any plan that destroys the Shipments table [2.79587475s]
[ci/terraform-checks] ⭐ Run Main Publish the plan to the job summary
[ci/terraform-checks]   ✅  Success - Main Publish the plan to the job summary [100.399167ms]
[ci/terraform-checks] ⭐ Run Main Upload the plan for apply.yml to use later
[ci/terraform-checks]   ✅  Success - Main Upload the plan for apply.yml to use later [1.086368541s]
[ci/terraform-checks] ⭐ Run Complete job
[ci/terraform-checks]   ✅  Success - Complete job
[ci/terraform-checks] 🏁  Job succeeded

Eleven steps, the same ci.yml as always, with one more step that changed nothing about the outcome: Plan: 12 to add, exactly like every earlier run of this guide on this same change. The guardrail doesn't interfere with any legitimate change — it's exactly the behavior you'd expect from a well-designed safety net: invisible when it isn't needed, decisive when it is.

Case 2 — The destruction attempt: the pipeline stops before applying

Reproduce lesson 7's complete scenario: seed the fixture, apply the destructive change, and run the test workflow.

cp .github/guardrail-fixtures/shipments-already-applied.state.json /tmp/seed.json

Empty out dynamodb.tf and remove the dependent output from outputs.tf, exactly as in lesson 7:

# dynamodb.tf — the Shipments table was decommissioned here by mistake

terraform validate

What to expect (literal):

Success! The configuration is valid.

rm -f terraform.tfstate
act workflow_dispatch -j destroy-shipments-check -W .github/workflows/guardrail-demo.yml

What to expect (literal output, executed to write this lesson):

[guardrail-demo/destroy-shipments-check] ⭐ Run Set up job
[guardrail-demo/destroy-shipments-check]   ✅  Success - Set up job
[guardrail-demo/destroy-shipments-check] ⭐ Run Main Check out andes-cargo-infra
[guardrail-demo/destroy-shipments-check]   ✅  Success - Main Check out andes-cargo-infra [69.198542ms]
[guardrail-demo/destroy-shipments-check] ⭐ Run Main Set up Terraform
[guardrail-demo/destroy-shipments-check]   ✅  Success - Main Set up Terraform [2.42054s]
[guardrail-demo/destroy-shipments-check] ⭐ Run Main Terraform init
[guardrail-demo/destroy-shipments-check]   ✅  Success - Main Terraform init [15.786407458s]
[guardrail-demo/destroy-shipments-check] ⭐ Run Main Seed the state with the fixture (Shipments already exists)
[guardrail-demo/destroy-shipments-check]   ✅  Success - Main Seed the state with the fixture (Shipments already exists) [776.935083ms]
[guardrail-demo/destroy-shipments-check] ⭐ Run Main Install tflocal
[guardrail-demo/destroy-shipments-check]   ✅  Success - Main Install tflocal [6.340297833s]
[guardrail-demo/destroy-shipments-check] ⭐ Run Main Terraform plan (offline, against the seeded fixture)
[guardrail-demo/destroy-shipments-check]   | Plan: 10 to add, 0 to change, 1 to destroy.
[guardrail-demo/destroy-shipments-check]   ✅  Success - Main Terraform plan (offline, against the seeded fixture) [6.640812125s]
[guardrail-demo/destroy-shipments-check] ⭐ Run Main Guardrail — block any plan that destroys the Shipments table
[guardrail-demo/destroy-shipments-check]   ❗  ::error::Guardrail failed: this plan destroys aws_dynamodb_table.shipments (Shipments). Blocking before apply.
[guardrail-demo/destroy-shipments-check]   ❌  Failure - Main Guardrail — block any plan that destroys the Shipments table [2.691965959s]
[guardrail-demo/destroy-shipments-check] ⭐ Run Complete job
[guardrail-demo/destroy-shipments-check]   ✅  Success - Complete job
[guardrail-demo/destroy-shipments-check] 🏁  Job failed
Error: Job 'destroy-shipments-check' failed

Plan: 10 to add, 0 to change, 1 to destroy. followed by 🏁 Job failed — the exact same sequence from lesson 7, run again here as the project's final confirmation: Andes Cargo's safety net works under pressure, not just in the isolated example where it got built. Notice what didn't happen, which matters just as much as what did: there was no Upload the plan for apply.yml to use later step — this blocked plan never became a downloadable artifact, so there isn't even a technical possibility of a mistakenly triggered apply.yml applying it.

Restoring the project

git checkout -- dynamodb.tf outputs.tf
rm -f terraform.tfstate tfplan plan.json
git status

What to expect (literal):

On branch main
nothing to commit, working tree clean

Module 6 close

You completed the module that teaches what to do when something goes wrong. Review what you're taking with you:

  • Infrastructure rollback, run end-to-end: git revert --no-edit on a real commit, with the complete history preserved (the original commit and its revert, both visible forever), ci.yml calculating the revert's plan, and apply.yml attempting to apply it with the same SHA256 matching between jobs you already saw in Module 5 (lessons 2-3).
  • Branch protection, with the exact click-path: the repository configuration —not a workflow— that turns "anyone can touch main" into "nothing lands without going through ci.yml," with the complete honesty about why act can't execute it (lesson 4).
  • The Claude Code incident, revisited in depth: traced through this module's five layers, with the honest two-part answer —no process control changes a bad human decision already made with the right information visible, but an automated guardrail makes some decisions technically impossible to execute, without depending on anyone's judgment (lesson 5).
  • conftest and policy-as-code, precisely named: the exact difference between a hand-crafted grep and a declarative policy evaluated against a data structure, with a clear boundary toward cloud-security-and-guardrails-guide (lesson 6).
  • A real guardrail, built and tested twice: a grep over terraform show -json, permanently integrated into ci.yml, that passes without interfering on a normal change and genuinely blocks —with literal output, red job— an attempt to destroy Shipments, including the honest state fixture technique to be able to test it without a real, completed apply (lesson 7 and this project).

What comes next

Module 7 locates everything you built in this guide within CI/CD's broader landscape: alternative tools (GitLab CI, CircleCI, Jenkins), Kubernetes's pull-based GitOps (ArgoCD, Flux), application deployment strategies, and the exact boundary between infrastructure CI/CD —what you built— and application code CI/CD. It doesn't add any new piece to andes-cargo-infra/ except a minimal contrast pipeline. Module 8, the capstone, runs the complete pipeline —ci.yml, apply.yml, drift.yml, and this module's guardrail— end-to-end, with a real change that does get approved and one the guardrail rejects, exactly like this project, but as part of the guide's final, complete walkthrough.

Common mistakes

Running Case 2 without restoring dynamodb.tf/outputs.tf before moving on to Module 7 (workflow-based, revisit lesson 7). What happens: someone runs this complete project, sees both cases work, and moves on without running the git checkout -- step. How to spot it: git status would show uncommitted changes, or the project would have the Shipments table permanently removed. How to fix it: always close this project —and lesson 7— with the project in a clean state, exactly as this lesson's restoration step shows.

Thinking this project tests something different from lesson 7 (scope-based). What happens: someone expects a new mechanism in this project, different from the guardrail already built. How to fix it: this project, on purpose, doesn't introduce any new mechanism — its value is exactly confirming the guardrail works integrated into the real pipeline, not just in the isolated example where it was first built. It's the same discipline you already saw in Module 5 (lesson 8): each piece built separately, run together at the end.

Exercises

Exercise 1 — Explain, to a colleague who only saw Case 1, why the guardrail "did nothing" in that run. Is it correct to say the guardrail didn't work in Case 1?

See solution

It's not correct to say it "didn't work" — the guardrail ran exactly the same way in both cases, evaluating the real plan against the same condition. In Case 1, the correct —and correctly calculated— evaluation is that there's no delete action on Shipments at all, so the guardrail passes and doesn't interfere. A guardrail that never lets anything through wouldn't be useful; one that evaluates correctly and distinguishes "this is safe" from "this is dangerous" is exactly what this project demonstrates in its two cases.

Exercise 2 — Reconstruct the exact reason apply.yml never ran in Case 2. Without looking at this lesson, explain the complete chain of events.

See solution

A complete answer sounds, roughly, like this: "The guardrail failed inside ci.yml's terraform-checks job (or, in this project, inside guardrail-demo.yml, which reproduces the same logic), with exit 1, before reaching the Upload the plan for apply.yml to use later step. Without that step, no artifact called terraform-plan ever gets uploaded. apply.yml depends, since Module 5, on downloading exactly that artifact to be able to apply anything — without it, not even the apply attempt could start, regardless of whether someone triggered apply.yml manually afterward."

Exercise 3 — Design a third test case, beyond this project's two. What other scenario, different from "harmless change" and "destroys Shipments," would be worth testing against this guardrail? Describe it and predict the outcome.

See solution

An interesting case: a plan that replaces the Shipments table —for example, changing hash_key to a different value, which forces Terraform to destroy it and create it again in the same plan ("actions":["delete","create"])—. The guardrail, as written, should block this case too: the grep searches for the word "delete" inside the actions array, regardless of whether it shows up alone or alongside "create" — a complete table replacement is, after all, exactly the kind of data loss this guardrail exists to prevent, even if it technically "recreates" the empty table afterward.

Summary and next step

In this project you ran Andes Cargo's complete pipeline with lesson 7's guardrail permanently integrated into ci.yml, confirmed in two real scenarios: a harmless change that went through all eleven steps with no interference at all (Plan: 12 to add, guardrail passing, artifact uploaded), and an attempt to destroy Shipments the pipeline stopped before any chance to apply (Plan: 10 to add, 1 to destroy, guardrail failing, red job, no artifact uploaded).

Before moving on you should be able to: describe from memory this project's complete walkthrough for both cases; explain why apply.yml never got the chance to run in Case 2; and confirm, in your own words, that the guardrail doesn't interfere with any legitimate Andes Cargo change.

With this, Module 6 is closed. You have infrastructure rollback tested end-to-end, branch protection understood with its exact click-path, the Claude Code incident revisited with the full depth Module 1 left pending, and a real guardrail, integrated and tested, protecting Andes Cargo's most critical resource.

Next module: the complete CI/CD landscape beyond this guide — GitLab CI, CircleCI, Jenkins by contrast, Kubernetes's pull-based GitOps, application deployment strategies, and the exact boundary between what you built (infrastructure CI/CD) and what's left for the sister guides (application CI/CD).

Resources

  1. nektosact.com — User Guide — complete reference for act pull_request and act workflow_dispatch, used end-to-end in this project.
  2. Terraform Internals — JSON Output Format — the JSON plan's schema, which the guardrail evaluates in both cases.
  3. This guide's Module 6 (07-hands-on-a-failing-guardrail-example.md) — the origin of the guardrail and the fixture this project reuses unchanged.
  4. This guide's Module 5 (08-project-andes-cargos-full-plan-to-apply-pipeline.md) — the sister project that ran ci.yml/apply.yml/drift.yml together for the first time, the same pattern this project extends with the guardrail.
← Previous7. Hands-on: a minimal, real, executed guardrailNext →1. Module 7 introduction: what this guide is not