Module 6: Supply Chain Sbom And Signing

8. Project: Andes Cargo's supply chain package

Next →

Description

Lessons 2 through 7 built four separate pieces: an SBOM (lessons 2-3), a local keypair (lessons 4-5), a verified signature (lesson 6), and proof that verification really detects tampering (lesson 7). This final project integrates them into what they really are from the first moment a pipeline uses them: a supply chain package, with a new job — verify-artifact — added to the apply.yml inherited from cicd-and-gitops-on-aws-guide, run with act end to end. You run it twice: once with the correct artifact (the gate lets it through), once with the altered artifact (the gate stops everything before terraform-apply even starts).

Connection to the module

This is the module's closing, and the same executed-evidence exercise every project in this guide has practiced since Module 1: not "did you generate an SBOM and a signature?", but "can you prove, with a pipeline really run, that an artifact without a valid signature never reaches apply?" RISK-MAP.md closes its TM-02 row here — the only one in the entire Tampering category across THREAT-MODEL.md.

Step 1 — The complete package, the four files

ls -la sbom.cyclonedx.json cosign.pub manifest.sig lambda/function.zip

What to expect (literal — sbom.cyclonedx.json/manifest.sig sizes may vary by a few bytes depending on metadata like the timestamp, already marked variable since lesson 3; function.zip is literal):

-rw-r--r--  4388  sbom.cyclonedx.json
-rw-r--r--   178  cosign.pub
-rw-r--r--   ~330 manifest.sig
-rw-r--r--   890  lambda/function.zip

Four files, four different, complementary roles: sbom.cyclonedx.json is the inventory (what's inside the project, including lesson 3's transitive dependencies); cosign.pub is the public key, the only piece a third party needs to verify any signature of yours; manifest.sig is the signature over the exact artifact; lambda/function.zip is the artifact the previous three pieces describe and protect, without being, itself, new to this module — it's the same .zip terraform-and-iac-guide generated. cosign.key, deliberately, doesn't appear in this list: it still exists on your local disk (.gitignore excludes it since lesson 5), but it's never part of what gets distributed nor of what a pipeline needs to verify — only to sign, a step that already happened, by hand, in lesson 6.

Step 2 — Extending apply.yml: the verify-artifact job

.github/workflows/apply.yml, inherited from cicd-and-gitops-on-aws-guide (Module 5, lesson 4), gains a new job, in parallel with fetch-reviewed-plan, and terraform-apply gains a second dependency:

name: apply

on:
  push:
    branches: [main]

jobs:
  fetch-reviewed-plan:
    runs-on: ubuntu-latest
    steps:
      - name: Download the plan reviewed in the pull request
        uses: actions/download-artifact@v4
        with:
          name: terraform-plan

      - name: Confirm the plan file arrived intact
        run: |
          test -s tfplan
          echo "tfplan is present: $(wc -c < tfplan) bytes"

  verify-artifact:
    runs-on: ubuntu-latest
    steps:
      - name: Check out andes-cargo-infra
        uses: actions/checkout@v4

      - name: Install envsubst (required by the cosign installer, missing on act's runner image)
        run: apt-get update -qq && apt-get install -y -qq gettext-base

      - name: Install cosign
        uses: sigstore/cosign-installer@6f9f17788090df1f26f669e9d70d6ae9567deba6

      - name: Verify function.zip against manifest.sig
        run: |
          cosign verify-blob \
            --key cosign.pub \
            --bundle manifest.sig \
            --insecure-ignore-tlog=true \
            lambda/function.zip

  terraform-apply:
    needs: [fetch-reviewed-plan, verify-artifact]
    runs-on: ubuntu-latest
    env:
      AWS_ACCESS_KEY_ID: test
      AWS_SECRET_ACCESS_KEY: test
      AWS_DEFAULT_REGION: us-east-1
      AWS_ENDPOINT_URL: http://host.docker.internal:4566
    steps:
      - name: Check out andes-cargo-infra
        uses: actions/checkout@v4

      - name: Set up Terraform
        uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: "1.15.8"

      - name: Download the plan reviewed in the pull request
        uses: actions/download-artifact@v4
        with:
          name: terraform-plan

      - name: Terraform init
        run: terraform init -input=false

      - name: Install tflocal
        run: pip3 install --quiet --break-system-packages terraform-local

      - name: Terraform apply
        run: tflocal apply -input=false -auto-approve tfplan

Three changes, and each deserves explaining:

First, the verify-artifact job itself — four steps, in order: check out the code (where cosign.pub, manifest.sig, and lambda/function.zip live, all three committed since earlier lessons), install a missing runner dependency (the next paragraph explains it), install cosign, and run exactly the same cosign verify-blob you ran by hand in lesson 6 — no change to the command, just inside a pipeline job instead of your terminal.

Second, a step no earlier lesson in this module needed: installing envsubst. While integrating sigstore/cosign-installer inside act for the first time, this guide found — and documents, instead of hiding — a real incompatibility between the medium runner image .actrc has used since cicd-and-gitops-on-aws-guide (catthehacker/ubuntu:act-latest) and that Action: cosign-installer's internal script depends on envsubst (part of Ubuntu's gettext-base package) to resolve a configurable install path, and that medium image — designed to be lightweight, not to replicate every tool a real GitHub runner has — doesn't include it by default, unlike GitHub's real runners, which do come with it preinstalled. Without this step, the job fails with envsubst: command not found — a lab infrastructure failure, not a flaw in this module's logic, and that's why it's fixed with one line, not hidden.

Third, needs: [fetch-reviewed-plan, verify-artifact] — before, terraform-apply only depended on fetch-reviewed-plan (from cicd-and-gitops-on-aws-guide, Module 5). Now it depends on both: the needs: mechanism with a list, which you already know from that module, requires all listed jobs to finish successfully before terraform-apply even starts running — not just one of them. It's, literally, this project's lock: if verify-artifact fails, terraform-apply never runs, no matter how perfectly fetch-reviewed-plan went.

Note on SHA pinning, revisiting lesson 1: sigstore/cosign-installer@6f9f17788090df1f26f669e9d70d6ae9567deba6 isn't pinned by tag (@v4) — it's pinned by the full commit SHA corresponding to tag v4.1.2, verified against GitHub's public API the same day this lesson was written (api.github.com/repos/sigstore/cosign-installer/tags). It's exactly the practice cicd-and-gitops-on-aws-guide (Module 2, lesson 5) named as the supply-chain security pillar it delegated to this guide: every third-party Action this module adds to the inherited pipeline is pinned by SHA, not a movable tag.

Step 3 — Running the gate with the correct artifact

act push -j verify-artifact -W .github/workflows/apply.yml

What to expect (literal output, run to write this lesson — filtered of act's repeated warnings that this lab directory isn't a Git repository, irrelevant to the result):

[apply/verify-artifact] ⭐ Run Set up job
[apply/verify-artifact] 🚀  Start image=catthehacker/ubuntu:act-latest
[apply/verify-artifact]   ✅  Success - Set up job
[apply/verify-artifact] ⭐ Run Main Check out andes-cargo-infra
[apply/verify-artifact]   ✅  Success - Main Check out andes-cargo-infra [11.675375ms]
[apply/verify-artifact] ⭐ Run Main Install envsubst (required by the cosign installer, missing on act's runner image)
[apply/verify-artifact]   | Setting up gettext-base (0.21-14ubuntu2) ...
[apply/verify-artifact]   ✅  Success - Main Install envsubst (required by the cosign installer, missing on act's runner image) [6.576651834s]
[apply/verify-artifact] ⭐ Run Main Install cosign
[apply/verify-artifact]   | INFO: Downloading bootstrap version 'v3.0.6' of cosign to verify version to be installed...
[apply/verify-artifact]   | INFO: bootstrap version successfully verified and matches requested version so nothing else to do
[apply/verify-artifact]   ✅  Success - Main Install cosign [4.038418125s]
[apply/verify-artifact] ⭐ Run Main Verify function.zip against manifest.sig
[apply/verify-artifact]   | WARNING: Skipping tlog verification is an insecure practice that lacks transparency and auditability verification for the blob.
[apply/verify-artifact]   | Verified OK
[apply/verify-artifact]   ✅  Success - Main Verify function.zip against manifest.sig [74.244958ms]
[apply/verify-artifact] ⭐ Run Complete job
[apply/verify-artifact]   ✅  Success - Complete job
[apply/verify-artifact] 🏁  Job succeeded

Job succeeded — and, at the center of the output, exactly the same Verified OK you already saw in lesson 6, now running inside an ephemeral Docker container, installed from scratch on every run, with the same result. This is, in itself, a second confirmation of the signature — it doesn't just work on your machine with the cosign you installed by hand in lesson 5, it works the same way in a completely different execution environment (catthehacker/ubuntu:act-latest, a clean Ubuntu container), with a different cosign install (v3.0.6, the bootstrap version the installer itself uses to verify itself, as the log shows) — the signature doesn't depend on any specific detail of your development environment.

Step 4 — Running the gate with the altered artifact: apply never comes into being

Repeat lesson 7's experiment, but this time inside the complete pipeline, not just with cosign by hand. Temporarily replace lambda/function.zip with an altered version (one extra byte, exactly as in lesson 7) and run the complete workflow, without scoping it to a single job:

cp lambda/function.zip.tampered lambda/function.zip   # simulates lesson 7's tampering
act push -W .github/workflows/apply.yml

What to expect (literal output, run to write this lesson — trimmed to the lines showing each job's result):

[apply/fetch-reviewed-plan] ❗  ::error::Unable to download artifact(s): Unable to get the ACTIONS_RUNTIME_TOKEN env variable
[apply/fetch-reviewed-plan] 🏁  Job failed
[apply/verify-artifact    ] ⭐ Run Main Verify function.zip against manifest.sig
[apply/verify-artifact    ]   | Error: failed to verify signature: could not verify message: invalid signature when validating ASN.1 encoded signature
[apply/verify-artifact    ]   ❌  Failure - Main Verify function.zip against manifest.sig [72.758625ms]
[apply/verify-artifact    ] 🏁  Job failed
Error: Job 'verify-artifact' failed

Not a single line from terraform-apply appears anywhere in this output. It didn't fail — it never got to run at all. needs: [fetch-reviewed-plan, verify-artifact] means Stage 1 (terraform-apply) only activates after both Stage 0 jobs finish successfully, and here both fail: verify-artifact for the exact reason this entire module exists (the signature doesn't match the altered artifact), and fetch-reviewed-plan for a completely different, already-familiar reason from cicd-and-gitops-on-aws-guide — this specific run didn't include the --artifact-server-path/--artifact-server-addr flags with a prior ci.yml that would have uploaded a real tfplan, so there's no terraform-plan artifact to download. Separate them carefully: this project's lesson is about the second failure (verify-artifact), not the first, which is simply the expected state of not having run the complete ci.yml first in this lab session.

Restore the correct artifact before continuing:

git checkout -- lambda/function.zip   # or: cosign verify-blob confirms "Verified OK" again after restoring

How to defend this work in an interview

A technical interviewer reviewing this project doesn't need you to recite the CycloneDX spec or the ECDSA algorithm from memory. They need you to be able to answer, without hesitating, three kinds of question:

  1. "Why SBOM and signature, not just one of the two?" — the answer lives in lesson 2: an SBOM without a signature tells you what's inside a package, but not whether the package is the one your team actually produced; a signature without an SBOM confirms the package didn't change, but not what it contains. This module builds both because they answer complementary questions, not the same question twice.

  2. "Why a local keypair and not keyless, if Sigstore offers both options?" — the answer lives in lesson 4, and has three concrete parts: the same real OIDC validation limit Module 2 already documented on LocalStack Hobby, the need for everything to run $0 without depending on a network connection to public services, and confirmation that a local keypair is still real Sigstore, not a reduced version. An interviewer from a team with real GitHub Actions and AWS would also understand that, in their context, keyless would probably be the right choice — the answer proves you know when each mechanism applies, not that you memorized "keypair is better."

  3. "How do you know this works, not just that it exists?" — the answer is this entire project: lesson 7 broke the signature on purpose and watched it fail with an explicit message; this project ran the complete gate twice, once with the correct artifact (Job succeeded, terraform-apply would have activated) and once with the altered one (Job failed, terraform-apply never ran) — never "it should block an unsigned artifact," always "it ran, and this is what happened, with the complete log as evidence."

Updating RISK-MAP.md: the sixth row closed

With TM-02 marked Resolved, RISK-MAP.md ends this module with six of seven rows closed — only TM-03 (CloudTrail, Module 7) remains open:

OrderIDControlStatus
1TM-01OIDC federation + scoped trust policyResolved (M2)
2TM-07Least-privilege role tighteningResolved (M2.7)
3TM-05SSM Parameter Store / Secrets ManagerResolved (M3)
4TM-04no-public-buckets.regoResolved (M4.7)
5TM-06no-destroy-shipments.regoResolved (M4.6)
6TM-02SBOM + cosign sign-blob/verify-blobResolved (M6.6, verified in pipeline in M6.8)
7TM-03CloudTrailOpen (Module 7)

The evidence note for this row, following the same discipline RISK-MAP.md has demanded since Module 1 — never just the word "Resolved" with nothing more: "lambda/function.zip signed with cosign sign-blob (local keypair, offline) in M6.6; verification confirmed Verified OK against the correct artifact and Error: invalid signature against a deliberately altered one in M6.7; both results reproduced inside a real verify-artifact job in apply.yml, run with act push, in M6.8 — terraform-apply stays gated by needs: on this verification passing."

The complete project, at a glance

  andes-cargo-infra/
  ├── THREAT-MODEL.md              (M1)
  ├── RISK-MAP.md                  (M1 → 6/7 rows Resolved by the close of M6)
  ├── secrets.tf                   (M3)
  ├── policy/                      (M4)
  ├── modules/
  │   ├── s3-bucket/
  │   ├── iam-role/
  │   └── oidc-provider/           (M2)
  ├── lambda/
  │   ├── handler.py                (inherited, 1801 bytes)
  │   ├── function.zip               (inherited, 890 bytes — THE artifact this module protects)
  │   └── requirements.txt          (M6.3 — boto3 + 6 resolved transitive dependencies)
  ├── sbom.cyclonedx.json           (M6.3 — 8 components)
  ├── cosign.pub                    (M6.5 — committed)
  ├── cosign.key                    (M6.5 — gitignored, NEVER versioned)
  ├── manifest.sig                  (M6.6 — the signature, offline, no Rekor)
  └── .github/workflows/
      └── apply.yml                 ← extended with verify-artifact (M6.8)

Common mistakes

Interpreting fetch-reviewed-plan failing in Step 4 as part of this module's lesson (attribution mistake, the most important one in this project). What happens: someone, seeing two jobs fail at once in Step 4, concludes this module also resolved something about downloading artifacts between ci.yml and apply.yml. How to spot it: if your explanation of Step 4's output mentions ACTIONS_RUNTIME_TOKEN as part of this module's thesis. How to fix it: that failure is inherited plumbing, unchanged, from cicd-and-gitops-on-aws-guide — it happens in this project only because Step 4 didn't first run a ci.yml with --artifact-server-path/--artifact-server-addr to produce a real tfplan to download, a step out of this module's scope. This project's lesson, specifically, is verify-artifact's: that failure really is new, really is intentional, and really is what this entire module built.

Leaving lambda/function.zip in its altered state after Step 4, without restoring it (project-hygiene mistake, revisits Module 4's pattern). What happens: someone finishes Step 4, sees the expected Job failed, and moves on without restoring the original artifact. How to spot it: if cosign verify-blob --key cosign.pub --bundle manifest.sig --insecure-ignore-tlog=true lambda/function.zip still fails after you thought you'd finished this project. How to fix it: every lesson in this module that introduced a test change (lesson 7, and this project) explicitly reverted that change before declaring the step complete — this project's final state, at closing, is the correct one, with Verified OK confirmed, not the demonstration's intermediate failure state.

Assuming adding envsubst as a step is a permanent patch any third-party Action is going to need (generalization mistake, over a case specific to this Action). What happens: someone, after resolving the envsubst problem for sigstore/cosign-installer, assumes any new Action they add to a workflow under act is going to need the same kind of preventive step. How to spot it: if you start adding "just in case" tool installs before even testing a new Action under act. How to fix it: this lesson's problem is specific to a concrete internal dependency of sigstore/cosign-installer (its install script uses envsubst) combined with a specific limitation of act's medium image (doesn't include it by default). Other Actions may depend on completely different tools, or none at all. The correct approach, every time, is to run the new Action under act first, read the real error if one appears, and fix the specific cause — exactly the process that produced this lesson's step, not a superstition applied in advance.

Exercises

Exercise 1 — Reconstruct, from memory, apply.yml's complete needs: chain after this module. Without looking at this lesson's YAML, sketch out (in text) which jobs exist, which run in Stage 0, which in Stage 1, and what that last one depends on.

See solution

Stage 0 (no needs:, run in parallel): fetch-reviewed-plan and verify-artifact. Stage 1 (needs: [fetch-reviewed-plan, verify-artifact]): terraform-apply, which only activates if both Stage 0 jobs finish successfully. Before this module, terraform-apply only depended on fetch-reviewed-plan (inherited from cicd-and-gitops-on-aws-guide); this project added verify-artifact as a second mandatory condition, not as an alternative — both have to pass, one of the two isn't enough.

Exercise 2 — Explain why this module's gate runs in Stage 0, alongside fetch-reviewed-plan, and not as a step inside terraform-apply itself. A colleague proposes simplifying the workflow by moving cosign verify-blob into one more step, inside terraform-apply, before tflocal apply. What would you lose with that change?

See solution

You'd lose the ability for verify-artifact to run in parallel with fetch-reviewed-plan (faster, in a pipeline with jobs that really take real minutes) and, more importantly, you'd lose the clarity that verification is an independent gate, with its own result visible in the GitHub Actions interface (a separate job, with its own success/failure icon), not a step hidden inside another job with a different purpose. Moving it inside terraform-apply would still block apply in practice (set -e would make the whole job fail if cosign verify-blob failed), but it would lose the separation of responsibilities: terraform-apply would stop being "the job that applies infrastructure" and become "the job that applies infrastructure and also verifies signatures," a mix of purposes that, over time, makes it harder to tell at a glance which of the two things failed.

Exercise 3 — Defend, against a concrete objection, why this module signs the deployment .zip and not, instead, each individual repository commit with git commit -S. A colleague experienced with Git asks why this guide doesn't use Git's native commit signing (git commit -S, verifiable with GPG or Git's own SSH/cosign signing support since recent versions) instead of signing the deployment artifact separately. How would you respond?

See solution

A complete answer distinguishes what each mechanism guarantees. Signing commits confirms that a specific commit, in Git's history, was created by who claims to have created it — a valuable guarantee about source code authorship, but one that says nothing about the deployment artifact a pipeline builds afterward: between the signed commit and the final .zip there are steps — checkout, packaging with archive_file, and in a more complex project, perhaps dependency installation or a build step — that a signed commit doesn't cover at all. An attacker who compromised the build process itself (the CI runner, for example) could produce a malicious .zip from a perfectly legitimate, signed commit, without the commit's signature detecting it. Signing the deployment artifact, as this module does, closes exactly that gap: it guarantees the .zip Lambda is going to execute is, byte for byte, the one that came out of the build process at the moment it was signed — a different, complementary guarantee to commit signing, not a substitute for it. A mature supply-chain system, in a real project, would probably use both layers, not just one.

Summary and next step

In this project you integrated this module's four pieces — SBOM, keypair, signature, verification proven against tampering — into a real verify-artifact job inside apply.yml, and ran the complete pipeline twice with act: once with the correct artifact (Job succeeded, the same Verified OK from lesson 6 confirmed inside an ephemeral container), and once with the altered artifact (Job failed, and terraform-apply never ran, gated by needs:). You documented, without hiding it, a real incompatibility between act's runner image and the official cosign install Action — and fixed it with one line, not a shortcut. You closed TM-02 from RISK-MAP.md, leaving the document with six of seven rows resolved.

Before closing this module you should be able to: run cosign sign-blob/verify-blob from scratch, without looking at any earlier lesson; explain, with this lesson's three interview questions, why SBOM and signature are complementary, why this guide chose keypair over keyless, and how you'd prove — with executed evidence, not a promise — that verification really works; and defend, against a colleague's objection, why signing the deployment artifact is a distinct, necessary layer alongside signing Git commits.

With this, cloud-security-and-guardrails-guide's Module 6 is complete: the market gap VALIDACION.md flagged as "zero" across the entire competition — SBOM, cosign/Sigstore, and Module 5's scanning — closed with executed evidence in every lesson, including cosign's real version drift documented live, not hidden. Module 7 opens this guide's last layer before the capstone: the distinction between preventive and detective guardrails, with TM-03 (CloudTrail) as the last THREAT-MODEL.md finding still to be resolved.

Resources

  1. Sigstore — Official cosign documentation — the complete sign-blob/verify-blob reference used throughout this module.
  2. This module, lessons 3, 5, 6, and 7 — the origin of each of the four pieces brought together in this project, with their individual evidence already verified.
  3. This course, Module 1, THREAT-MODEL.md/RISK-MAP.md — finding TM-02, which this project closes, and the row it updates.
  4. cicd-and-gitops-on-aws-guide, Module 2, lesson 5 and Module 5, lesson 4 — the SHA pinning and the original apply.yml this project extends, without rewriting it.
  5. nektosact.com — User Guide — the act push -W reference, used throughout this project to scope and run the extended workflow's runs.
← Previous7. Hands-on: breaking the chain on purposeNext →1. Introduction: the handrail vs. the security camera