Module 6: Rollback And Safety Nets

3. Hands-on: the `git revert` pipeline

Next →

Description

This is the lesson where lesson 2's mechanism stops being a diagram and becomes something you ran with your own hands: you're going to genuinely revert the commit that added the Compliance tag to the andes-cargo-shipment-docs bucket (Module 3, lesson 8), run ci.yml with act on that revert, confirm in the literal output that the tag disappeared from the plan, and run apply.yml to see the same real attempt —with the same honest failure as always— applying it.

Connection to the module

Lesson 2 gave you the complete mechanism, in theory. This lesson runs it on the same repository you've been building since Module 1: same ci.yml, same apply.yml, no new file except the PR event that simulates the revert. Lesson 4 changes topics —branch protection—, but the commit thread this lesson leaves in andes-cargo-infra/ stays, forever, part of the project's real history.

Starting point: the commit you're going to revert

Recall Module 3's close: the project added a Compliance = "manifest-retention-required" tag to the manifest bucket, with commit a10c9b7 Add Compliance tag to the shipment-docs bucket. Imagine that, weeks later, Andes Cargo's compliance team reports that retention policy still isn't formally approved — the tag got added ahead of schedule, and it needs to come off until the approval process finishes. It's exactly the kind of "business mistake" lesson 1 named: nothing technical went wrong, the apply (had it run against a real account) would have worked with no errors — the problem is that the change should never have been approved yet.

cd andes-cargo-infra
git log --oneline | grep -i compliance

What to expect (literal, the exact commit that added the tag in Module 3):

a10c9b7 Add Compliance tag to the shipment-docs bucket

Step 1 — git revert, for real

git revert --no-edit a10c9b7

--no-edit accepts the commit message Git generates automatically, without opening a text editor — perfectly acceptable here, because the generated message already precisely describes what happened.

What to expect (literal for the structure, variable for the hash — yours is going to be different from this run's, because it depends on the exact moment of the commit):

[main <hash>] Revert "Add Compliance tag to the shipment-docs bucket"
 Date: <date-from-your-machine>
 1 file changed, 1 insertion(+), 3 deletions(-)

git log --oneline -3

What to expect (literal for the structure and messages, <hash> variable):

<hash> Revert "Add Compliance tag to the shipment-docs bucket"
8d4df2c drift.yml: schedule-based drift detection with terraform_wrapper: false
24fe98f apply.yml: needs, artifact download, and the real terraform apply attempt

Notice something lesson 2 already flagged: a10c9b7 is still in your history, further down — it didn't disappear. Confirm it:

git log --oneline | grep -i compliance

What to expect (literal for the structure, <hash> variable in the first line):

<hash> Revert "Add Compliance tag to the shipment-docs bucket"
a10c9b7 Add Compliance tag to the shipment-docs bucket

Two commits, both visible, forever: the one that added the tag and the one that reverted it. Exactly the auditable history lesson 2 promised.

Confirm the exact diff the revert produced:

git show --stat HEAD
cat s3.tf

What to expect (literal — s3.tf back to the state before Module 3, lesson 8, with tags = local.common_tags instead of the merge(...) that added Compliance):

module "shipment_docs_bucket" {
  source = "./modules/s3-bucket"

  bucket_name        = var.bucket_name
  enable_versioning  = true
  bucket_policy_json = data.aws_iam_policy_document.require_https.json
  tags               = local.common_tags
}

Step 2 — ci.yml on the revert: the same plan, a different change

Simulate the Pull Request that would bring this revert up for review. Write a new event, with a PR number and branch name that honestly describe what it is —a revert, not a new feature—:

.github/act-events/pr-event-revert.json:

{
  "action": "opened",
  "number": 45,
  "pull_request": {
    "number": 45,
    "title": "Revert: Add Compliance tag to the shipment-docs bucket",
    "head": {
      "ref": "revert-shipment-tags",
      "sha": "b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1"
    },
    "base": {
      "ref": "main",
      "sha": "1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b"
    },
    "html_url": "https://github.com/andes-cargo/andes-cargo-infra/pull/45",
    "user": {
      "login": "andes-cargo-dev"
    }
  },
  "repository": {
    "name": "andes-cargo-infra",
    "full_name": "andes-cargo/andes-cargo-infra",
    "default_branch": "main"
  }
}

sha in head/base are hand-made-up values, exactly like Module 2's original pr-event.json — they don't represent any real GitHub commit, they just need the correct format of a Git SHA for the payload to be valid.

Run the same ci.yml as always, without changing a single line of that file:

export ARTIFACT_ADDR=$(ipconfig getifaddr en0)   # Linux: hostname -I | awk '{print $1}'
rm -rf .artifacts && mkdir -p .artifacts

act pull_request -e .github/act-events/pr-event-revert.json -j terraform-checks \
  --artifact-server-path ./.artifacts \
  --artifact-server-addr "$ARTIFACT_ADDR"

What to expect (literal output, executed to write this lesson):

[ci/terraform-checks] ⭐ Run Set up job
[ci/terraform-checks]   ✅  Success - Set up job
[ci/terraform-checks] ⭐ Run Main Check out andes-cargo-infra
[ci/terraform-checks]   ✅  Success - Main Check out andes-cargo-infra [50.523167ms]
[ci/terraform-checks] ⭐ Run Main Set up Terraform
[ci/terraform-checks]   ✅  Success - Main Set up Terraform [2.533139042s]
[ci/terraform-checks] ⭐ Run Main Terraform format check
[ci/terraform-checks]   ✅  Success - Main Terraform format check [139.099541ms]
[ci/terraform-checks] ⭐ Run Main Terraform init
[ci/terraform-checks]   ✅  Success - Main Terraform init [14.141802916s]
[ci/terraform-checks] ⭐ Run Main Terraform validate
[ci/terraform-checks]   ✅  Success - Main Terraform validate [2.080303708s]
[ci/terraform-checks] ⭐ Run Main Install awslocal
[ci/terraform-checks]   ✅  Success - Main Install awslocal [8.526287s]
[ci/terraform-checks] ⭐ Run Main Confirm the runner can reach LocalStack on the host
[ci/terraform-checks]   | Could not connect to the endpoint URL: "http://host.docker.internal:4566/"
[ci/terraform-checks] Failed but continue next step
[ci/terraform-checks]   ❌  Failure - Main Confirm the runner can reach LocalStack on the host [13.07124875s]
[ci/terraform-checks] ⭐ Run Main Install tflocal
[ci/terraform-checks]   ✅  Success - Main Install tflocal [2.301586834s]
[ci/terraform-checks] ⭐ Run Main Terraform plan
[ci/terraform-checks]   | Plan: 12 to add, 0 to change, 0 to destroy.
[ci/terraform-checks]   ✅  Success - Main Terraform plan [4.722843583s]
[ci/terraform-checks] ⭐ Run Main Publish the plan to the job summary
[ci/terraform-checks]   ✅  Success - Main Publish the plan to the job summary [73.929ms]
[ci/terraform-checks]   ⚙  Summary - ## Terraform plan — andes-cargo-infra
[ci/terraform-checks] ⭐ Run Main Upload the plan for apply.yml to use later
[ci/terraform-checks]   | Artifact terraform-plan has been successfully uploaded! Final size is 12428 bytes.
[ci/terraform-checks]   ✅  Success - Main Upload the plan for apply.yml to use later [909.815333ms]
[ci/terraform-checks] ⭐ Run Complete job
[ci/terraform-checks]   ✅  Success - Complete job
[ci/terraform-checks] 🏁  Job succeeded

Reading the revert's plan: why it still says "12 to add"

Before celebrating that the revert "worked," read this number with the same care you already learned in Module 5 (lesson 7). Plan: 12 to add, 0 to change, 0 to destroy. is identical to any other run of this project on this machine — it doesn't say 1 to change, as you'd expect from a revert that only removes a tag from an already-existing resource. The reason is the same as always: this project's state, without a real apply completed against LocalStack, is still completely empty. A plan on an empty state always calculates a complete creation of the 12 resources, no matter how small the HCL change that triggered it was — the revert is real, the HCL change is real, but the plan has no existing resource to compare that change against as a simple update.

What you can confirm, with the same discipline Module 3 (lesson 8) already taught you, is that the plan's content reflects the revert:

grep -c "Compliance" plan-output.txt

What to expect (literal): 0 — no occurrences, unlike the two occurrences you saw in Module 3 (lesson 8) before the revert.

Confirm it by looking at the bucket's complete tags block inside the plan:

  # module.shipment_docs_bucket.aws_s3_bucket.this will be created
  + resource "aws_s3_bucket" "this" {
      ...
      + tags                        = {
          + "Environment" = "dev"
          + "ManagedBy"   = "terraform"
          + "Project"     = "andes-cargo"
        }

Three tags, not four — Compliance disappeared. This is the real evidence the revert undid the change, even though the last line's number (12 to add) doesn't say so by itself. The underlying lesson, once again: read the plan's content, not just its summary, when the summary alone can't distinguish two different scenarios.

Step 3 — apply.yml: the same real attempt, the same honest failure

On a real repository, someone would review this plan —seeing that the Compliance tag disappears, exactly what PR #45's title promised— and approve the merge. Simulate it by running apply.yml directly, exactly like in Module 5 (lesson 4):

act push -W .github/workflows/apply.yml \
  --artifact-server-path ./.artifacts \
  --artifact-server-addr "$ARTIFACT_ADDR"

What to expect (literal output, executed to write this lesson — Stage 0 succeeding, the SHA256 matching what ci.yml uploaded):

[apply/fetch-reviewed-plan] ⭐ Run Main Download the plan reviewed in the pull request
[apply/fetch-reviewed-plan]   | SHA256 digest of downloaded artifact is 54e5f9fcdc6181197394e0f187aaa717e06d0423843dea623878c9962f9cbd50
[apply/fetch-reviewed-plan]   ✅  Success - Main Download the plan reviewed in the pull request [422.849417ms]
[apply/fetch-reviewed-plan] ⭐ Run Main Confirm the plan file arrived intact
[apply/fetch-reviewed-plan]   ✅  Success - Main Confirm the plan file arrived intact [67.458375ms]
[apply/fetch-reviewed-plan] 🏁  Job succeeded
[apply/terraform-apply    ] ⭐ Run Main Check out andes-cargo-infra
[apply/terraform-apply    ]   ✅  Success - Main Check out andes-cargo-infra [40.789375ms]
[apply/terraform-apply    ] ⭐ Run Main Set up Terraform
[apply/terraform-apply    ]   ✅  Success - Main Set up Terraform [2.383532167s]
[apply/terraform-apply    ] ⭐ Run Main Download the plan reviewed in the pull request
[apply/terraform-apply    ]   | SHA256 digest of downloaded artifact is 54e5f9fcdc6181197394e0f187aaa717e06d0423843dea623878c9962f9cbd50
[apply/terraform-apply    ]   ✅  Success - Main Download the plan reviewed in the pull request [424.188333ms]
[apply/terraform-apply    ] ⭐ Run Main Terraform init
[apply/terraform-apply    ]   ✅  Success - Main Terraform init [15.040384667s]
[apply/terraform-apply    ] ⭐ Run Main Install tflocal
[apply/terraform-apply    ]   ✅  Success - Main Install tflocal [5.5976365s]
[apply/terraform-apply    ] ⭐ Run Main Terraform apply
[apply/terraform-apply    ]   | aws_dynamodb_table.shipments: Creating...
[apply/terraform-apply    ]   | module.app_server_role.aws_iam_role.this: Creating...
[apply/terraform-apply    ]   | module.lambda_manifest_processor_role.aws_iam_role.this: Creating...
[apply/terraform-apply    ]   | module.shipment_docs_bucket.aws_s3_bucket.this: Creating...
[apply/terraform-apply    ]   | module.shipment_docs_bucket.aws_s3_bucket.this: Still creating... [00m40s elapsed]
[apply/terraform-apply    ]   | ╷
[apply/terraform-apply    ]   | │ Error: creating S3 Bucket (andes-cargo-shipment-docs): operation error S3: CreateBucket, exceeded maximum number of attempts, 9, ... connect: connection refused
[apply/terraform-apply    ]   | ╵
[apply/terraform-apply    ]   ❗  ::error::Terraform exited with code 1.
[apply/terraform-apply    ]   ❌  Failure - Main Terraform apply [53.744650083s]
[apply/terraform-apply    ] 🏁  Job failed
Error: Job 'terraform-apply' failed

The SHA256 matches across both jobs (54e5f9fcdc6181197394e0f187aaa717e06d0423843dea623878c9962f9cbd50), the same integrity proof from Module 5: what terraform-apply tried to apply is exactly the file ci.yml calculated, with no reprint in between. The final failure is, once again, the same as always —connection refused after 9 retries from the AWS provider, because this machine doesn't have LocalStack running with a valid token—: the pipeline's mechanism is real end-to-end; the only representative part, as in every previous module, is the final result of writing to infrastructure that doesn't exist here.

With a valid LOCALSTACK_AUTH_TOKEN and Andes Cargo's infrastructure already applied, this same apply would have ended in Apply complete!, with the andes-cargo-shipment-docs bucket back to just three tags — the complete rollback, end-to-end, without anyone touching the AWS console or running terraform apply by hand.

Common mistakes

Expecting 1 to change in the revert's plan (expectation-based, this lesson's central mistake). What happens: someone runs the revert, sees Plan: 12 to add, and concludes the revert didn't work. How to spot it: you compare the number against what you expected from lesson 2 (an update, not a creation) and it doesn't match. How to fix it: as this lesson already explained, the last line's number depends on the state, which stays empty on this machine — the revert did work, and the correct evidence is the plan's content (grep -c "Compliance" at 0), not the last line's summary.

Forgetting a10c9b7 is still in the history after the revert (conceptual, revisit lesson 2). What happens: someone looks for the original commit with git log and, not finding it in the first few lines, assumes it disappeared. How to fix it: git log --oneline with no limit (or git log --oneline | grep -i compliance) shows both commits — the original is still exactly where it was, the revert just got added after it, never replacing it.

Confusing apply.yml's failure with a problem with the revert (diagnosis-based, revisit Module 5). What happens: someone sees red Job failed after the revert and suspects something went wrong specifically with this change. How to spot it: the message —connection refused after real retries, on the four root resources— is identical, character for character, to what you already saw in Module 5 (lesson 4) for a completely different change. How to fix it: docker ps -a --filter name=localstack_main confirms, once again, that the cause is the same as always —LocalStack isn't running—, with no relation to this specific revert's content.

Exercises

Exercise 1 — Reconstruct the complete commit thread from memory. Without looking at this lesson, describe in order the two commits that now exist regarding the Compliance tag, and explain which Git command produced each one.

See solution

First, a10c9b7 Add Compliance tag to the shipment-docs bucket (Module 3, lesson 8) — a normal, hand-written commit, that added the tag via merge(local.common_tags, { Compliance = "..." }). Then, a new commit (variable hash) with the message Revert "Add Compliance tag to the shipment-docs bucket" — produced by git revert --no-edit a10c9b7, whose content is exactly the opposite diff: it removes the merge(...) and leaves tags = local.common_tags. Both commits coexist, in that order, in main's history.

Exercise 2 — Explain why the revert's plan doesn't say 1 to change, to a colleague expecting to see it. Use this lesson's and Module 5's exact vocabulary.

See solution

A complete answer sounds, roughly, like this: "~ update in-place (the symbol that would produce 1 to change) only shows up when Terraform's state already has the resource registered as existing, and the HCL proposes modifying one of its attributes. On this machine, without a valid LOCALSTACK_AUTH_TOKEN, we never completed a real apply, so the state stays empty — Terraform has no 'existing' bucket to compare the reverted HCL against, so it calculates, once again, a complete creation of the 12 resources. The revert is real and is reflected in the plan's content —the Compliance tag no longer shows up anywhere—, it's just that the last line's summary can't distinguish 'creation from scratch, without the tag' from 'update that removes the tag,' because both cases start from the same final HCL."

Exercise 3 — Design the commit that would revert this revert. If Andes Cargo decided, the following week, that the Compliance tag actually was approved after all and needed to be added back, what Git command would you use, and what would happen to the history?

See solution

git revert --no-edit <revert-commit-hash> — reverting the revert, with this lesson's exact same mechanism, now pointing at the "Revert..." commit's hash. The result would be a third new commit, whose content adds the Compliance tag back — the complete history would then have three commits related to this tag, all visible, all with their own author and date: the original, the revert, and the revert of the revert. None of the three ever disappears; the history simply grows, becoming more precise each time about what happened and when.

Summary and next step

In this lesson you ran infrastructure rollback's complete mechanism, end-to-end: git revert --no-edit a10c9b7 created a new commit without erasing the original, ci.yml ran the same plan as always on that revert (with Plan: 12 to add unchanged, but with the Compliance tag genuinely absent from the content), and apply.yml genuinely attempted to apply it, with the same SHA256 matching between jobs and the same honest connection failure you already know from Module 5.

Before moving on you should be able to: run git revert --no-edit on any commit in this project; explain why a revert's plan, on this specific machine, doesn't change the "to add" resource count; and confirm a revert's real content with grep, without relying only on a plan's last-line summary.

Lesson 4 changes layers entirely: instead of a mechanism running inside a workflow, you're going to see the control that lives in the repository's own configuration —branch protection— and why no act can execute it, even though its effect on main is completely real.

Resources

  1. Git Docs — git revert — official reference, including --no-edit.
  2. nektosact.com — User Guide — reference for act pull_request -e and act push -W, used end-to-end in this lesson.
  3. This guide's Module 3 (08-project-andes-cargos-ci-workflow.md) — the Compliance tag's original commit, which this lesson reverts.
  4. This guide's Module 5 (04-hands-on-building-apply-yml.md) — the SHA256 verification mechanism and the honest terraform apply failure pattern, reused here unchanged.
← Previous2. What rollback means for infrastructureNext →4. Branch protection as a control