Module 6: Supply Chain Sbom And Signing
7. Hands-on: breaking the chain on purpose
Description
Lesson 6's Verified OK confirms the signature works when nothing changes. This lesson tests the other half of the guarantee, the one that really matters in a real incident: what happens when something does change? You're going to modify lambda/function.zip after signing it — a single byte appended to the end of the file, nothing more — and run cosign verify-blob against that altered version. The signature is going to fail, live, with an explicit error message. And you're going to discover, along the way, an uncomfortable, real fact: the altered .zip file is still a perfectly valid .zip, which unzip opens without any complaint — the cryptographic signature detects something the .zip format's own validation doesn't detect at all.
Connection to the module
This lesson is lesson 6's necessary counterpart: a signature that's never been seen to fail isn't a signature you can trust, it's a signature you hope works. Lesson 8 is going to use this exact same mechanism — cosign verify-blob returning a non-zero exit code — as the condition that stops an apply inside the inherited pipeline.
Step 1 — Confirming the clean starting state
Before breaking anything, confirm lesson 6's manifest.sig still verifies correctly against the unmodified lambda/function.zip — the reference point you'll compare the rest of this lesson against:
cosign verify-blob --key cosign.pub --bundle manifest.sig --insecure-ignore-tlog=true lambda/function.zip
echo "exit: $?"What to expect (literal):
WARNING: Skipping tlog verification is an insecure practice that lacks transparency and auditability verification for the blob.
Verified OK
exit: 0
This is the result you already saw in lesson 6. From here on, any change in the result has exactly one possible cause: what you do to the file, not something intermittent in the tool.
Step 2 — Modifying the artifact after signing, a single byte
Work on a copy — never on the real file you already signed, so you don't lose lesson 6's clean state — and append a single null byte to the end:
cp lambda/function.zip lambda/function.zip.tampered
printf '\x00' >> lambda/function.zip.tampered
ls -la lambda/function.zip lambda/function.zip.tamperedWhat to expect (literal, run to write this lesson):
-rw-r--r-- 890 lambda/function.zip
-rw-r--r-- 891 lambda/function.zip.tampered
A one-byte difference — 890 versus 891. You didn't rewrite handler.py, didn't recompress anything, didn't touch the content Lambda would execute when unpacking the file: you only appended one byte to the end of the already-built .zip. It's, deliberately, the smallest possible alteration — if the signature catches this, it catches anything bigger too.
Step 3 — The uncomfortable finding: the altered .zip is still a valid .zip
Before verifying the signature, it's worth confirming something that surprises people the first time: the .zip format, by its own internal structure, tolerates extra bytes at the end of the file without considering it corrupt. Confirm it:
unzip -t lambda/function.zip.tamperedWhat to expect (literal, run to write this lesson):
Archive: lambda/function.zip.tampered
testing: handler.py OK
No errors detected in compressed data of lambda/function.zip.tampered.
Zero errors. unzip opens the file, decompresses handler.py, and reports it as intact — because, technically, it is: the .zip format stores its complete index (the End of Central Directory) and looks for that structure starting from the end of the file backward; one extra byte after that structure breaks nothing unzip needs to work. If you uploaded this altered .zip to Lambda as-is, the function would probably still deploy and run with no visible error — the same handler.py, byte for byte, is still inside, unchanged.
WHAT unzip SEES WHAT cosign SEES
──────────────── ─────────────────
[ZIP header] [compressed handler.py] [ZIP header] [compressed handler.py]
[central directory] [EOCD] [central directory] [EOCD] [extra byte]
↑
unzip looks for the EOCD from the end cosign hashes the ENTIRE FILE,
and finds it — reports "OK" byte by byte, with no exceptionThis is exactly why this module exists, stated with the most concrete evidence possible: a file's format validity is not the same as its content's integrity. Trivy and Checkov (Module 5) can tell you whether your infrastructure configuration follows good practices; unzip -t can tell you whether a .zip is well-formed; neither one can tell you whether that file is, exactly, the one someone with authority approved. That's, uniquely, the question a signature answers.
Step 4 — Verifying the signature against the altered file
cosign verify-blob --key cosign.pub --bundle manifest.sig --insecure-ignore-tlog=true lambda/function.zip.tampered
echo "exit: $?"What to expect (literal, run to write this lesson):
WARNING: Skipping tlog verification is an insecure practice that lacks transparency and auditability verification for the blob.
Error: failed to verify signature: could not verify message: invalid signature when validating ASN.1 encoded signature
error during command execution: failed to verify signature: could not verify message: invalid signature when validating ASN.1 encoded signature
exit: 1
This is this version of cosign's real message — a bit different, in its exact wording, from a plain, generic "Error: invalid signature", and it's worth reading precisely instead of memorizing an approximate phrase: invalid signature when validating ASN.1 encoded signature — ASN.1 (Abstract Syntax Notation One) is the standard format the two numbers that make up an ECDSA signature are encoded in; cosign is saying, precisely, that it correctly decoded the signature's structure (it isn't a corrupt or malformed manifest.sig file), but that when it applied the mathematical verification operation against the hash of the file you gave it, that signature doesn't correspond to that hash. And the exit code — 1, not 0 — is the signal any script or CI job needs to stop what comes next.
Step 5 — Why this happens: the hash changed, the signature couldn't change with it
The exact technical reason, no magic involved: cosign sign-blob doesn't sign "the .zip" in some abstract sense — it signs the SHA-256 hash of the file's complete content, byte for byte, at the exact moment of signing. Confirm it yourself:
shasum -a 256 lambda/function.zip | awk '{print $1}' | xxd -r -p | base64
shasum -a 256 lambda/function.zip.tampered | awk '{print $1}' | xxd -r -p | base64What to expect (literal, run to write this lesson):
mX7XF7LjI00UJ+DFC6+/IXEoh/Cdsb9EOPAWXwmwn1U=
Ol85F3RXsWjhOQoIQzwvGKB5r5oqMDNVWaKlxDPq/6E=
A one-byte difference in the file produced a completely different hash — not "similar," not "almost the same" — a fundamental property of any well-designed cryptographic hash function, called the avalanche effect: the smallest possible change in the input produces a totally different, unpredictable output, never one "close" to the original. manifest.sig contains a signature calculated over the first hash (mX7XF7...); when cosign verify-blob recalculates the hash of the file you gave it (Ol85F3..., over the altered .zip) and compares it against what the signature mathematically guarantees, the two don't match, and verification fails — exactly the mechanism you just observed live in Step 4.
Step 6 — Restoring the correct state
rm lambda/function.zip.tampered
cosign verify-blob --key cosign.pub --bundle manifest.sig --insecure-ignore-tlog=true lambda/function.zipWhat to expect (literal — back to Step 1's result):
WARNING: Skipping tlog verification is an insecure practice that lacks transparency and auditability verification for the blob.
Verified OK
lambda/function.zip, the real file this module signed, was never touched — you only ever worked on a copy (function.zip.tampered), which you delete here. By the end of this lesson, andes-cargo-infra/ is exactly in the same correct state lesson 6 left it in.
An honest question: couldn't an attacker just re-sign the altered file?
It's worth precisely answering the most reasonable objection this lesson can raise: if someone manages to alter function.zip somewhere in the chain, couldn't they just generate a new signature over the altered file, and pass it off as legitimate? The answer, with the same honesty that governs this guide, has two parts. No, not without cosign.key: signing a new file requires the private key, which lesson 5's Step 5 explicitly excluded from the repository with .gitignore — an attacker who only has access to andes-cargo-infra/'s source code (for example, through a Git repository leak) doesn't have that key, and can't produce a valid signature over any file, altered or not. Yes, on the other hand, if the attacker also compromises the machine or process that does have access to cosign.key — for example, a compromised CI runner with the key loaded as a secret —: in that scenario, the signature alone protects nothing, because the attacker can really sign with the legitimate key. This is, precisely, why this module's signature is a defense layer within a broader system, not an isolated silver bullet: it protects against alteration of the artifact after it leaves the trusted build process, but it doesn't replace the need to protect that build process itself — the same least-privilege and federated-identity principle Modules 2 and 3 already built for the pipeline itself.
Common mistakes
Confusing the invalid-signature error with a corrupt .zip file (diagnosis mistake, the central one this lesson prevents). What happens: someone sees Error: failed to verify signature and assumes the .zip itself is damaged or malformed, and tries to "repair" it with ZIP file repair tools. How to spot it: if your first instinct facing this error is to run unzip -t or a repair tool, instead of asking whether the file changed since it was signed. How to fix it: this lesson's Step 3 already demonstrated it — a .zip can be perfectly well-formed (unzip -t reports zero errors) and still fail signature verification. They're two completely different questions: "is this a valid .zip file?" (answered by unzip) and "is this exactly the file someone signed?" (answered only by cosign verify-blob). A truly corrupt .zip would fail unzip -t first; a well-formed .zip altered after being signed only fails signature verification.
Thinking the "ASN.1 encoded signature" message indicates a problem with manifest.sig's format, not with function.zip's content (error-message-reading mistake). What happens: someone reads "ASN.1" — an unfamiliar term — and suspects the signature file itself is corrupt or poorly generated. How to spot it: if your first diagnostic step is to regenerate manifest.sig from scratch, before checking whether function.zip changed. How to fix it: as Step 4 explained, ASN.1 is just the standard encoding format for the numbers making up an ECDSA signature — the message says cosign decoded the signature correctly, but that the mathematical verification result doesn't match the file you gave it. The cause is almost always in the file being verified, not the signature itself — unless manifest.sig was also altered, a case that would produce a different error, about bundle parsing, not an invalid signature.
Assuming the signature protects the .zip's "logical" content (the files it contains) instead of the complete binary file (incorrect mental model). What happens: someone assumes cosign somehow understands the .zip's internal structure — what files it contains, with what content — and signs that, such that reordering internal files without changing their content wouldn't break the signature. How to spot it: if you expect two .zip files with the same handler.py but compressed with different parameters (for example, a different compression level) to verify against the same signature. How to fix it: cosign sign-blob/verify-blob have no knowledge of the .zip format at all — they treat the file as an arbitrary sequence of bytes ("blob," the command's own name says so) and sign the hash of those exact bytes, regardless of what format they represent. Any change to the resulting binary file — including recompression with different parameters, even if the logical content is identical — produces a different hash and breaks verification, exactly as one extra byte at the end broke it in this lesson.
Exercises
Exercise 1 — Predict the result if you altered cosign.pub instead of lambda/function.zip. Without running it, would you expect Step 4's same error message, a different one, or the command to fail in a completely different way?
See solution
A different, though related, message. Altering cosign.pub — for example, changing a single character inside the PEM block — would probably produce a public key parsing error (something like "failed to parse public key" or similar), not Step 4's "invalid signature" error, because cosign would fail before it even reached the mathematical verification operation: it wouldn't be able to interpret the file as a valid ECDSA key at all. If the change were subtle enough to still be a mathematically valid ECDSA key (but different from the original one used to sign), the result would look more like Step 4: a verification that runs to completion, but fails, because the public key no longer corresponds to the private one that produced the original signature.
Exercise 2 — Explain, in your own words, why unzip -t reporting "zero errors" on a file with an extra byte is NOT a bug in unzip. A colleague, surprised by Step 3's result, asks whether unzip should consider that a corrupt file. What would you tell them?
See solution
It isn't a bug — it's a reasonable consequence of the .zip format's design, which was built to tolerate a certain amount of extra data (for example, a self-extracting .zip deliberately has an executable stuck before the real ZIP structure, and it's still a perfectly valid .zip for any tool that opens it). The format looks for its index (End of Central Directory) starting from the end of the file, and as long as that search finds the expected structure, the rest gets ignored. unzip -t correctly answers the question it's meant to answer ("can I extract the declared content with no error?") — that simply isn't the same question as "is this file, byte for byte, identical to the one someone approved?", which is exactly the question this lesson demonstrated only a cryptographic signature answers.
Exercise 3 — Design, in prose, a third tampering scenario different from this lesson's, and predict whether cosign verify-blob would catch it. Instead of appending a byte at the end, imagine someone replaces handler.py inside the .zip with a malicious version, but recompresses the file so the resulting .zip has exactly the same byte size as the original (890 bytes). Would that change the verification result?
See solution
No, the result would be the same: Error: failed to verify signature. The altered file's byte size is irrelevant to verification — what matters is the SHA-256 hash over the complete binary content, and practically any change to the content (appending a byte, changing a single line of handler.py and recompressing, or any other alteration) produces, through the avalanche effect already explained in Step 5, a completely different hash — regardless of whether the resulting size happens to match the original. This exercise is, in fact, the more realistic and dangerous of the two tampering scenarios: a sophisticated attacker trying to hide their change by keeping the file size constant would still be detected with the same certainty as this lesson's much more obvious extra byte.
Summary and next step
In this lesson you broke the chain of trust on purpose, in the smallest possible way — a single byte appended to the end of lambda/function.zip after signing it — and confirmed, with cosign verify-blob's literal output, that verification fails with an explicit message and a non-zero exit code. You discovered, with real evidence, that the .zip format itself doesn't detect this alteration (unzip -t still reports "zero errors"), which makes the cryptographic signature the only layer in this module capable of answering the question that really matters: is this, exactly, the artifact someone approved? And you answered, honestly, exactly how far that protection reaches: against alteration after the signing point, yes; against an attacker who also had access to cosign.key, no — one more reason Module 2's identity and Module 3's secrets remain necessary, not replaced by this module.
Before moving on you should be able to: explain why an altered .zip can still be valid according to unzip and still fail signature verification; read the message invalid signature when validating ASN.1 encoded signature without confusing it with a corrupt file; and precisely defend how far a signature's guarantee reaches and where it stops.
Lesson 8 integrates everything this module built — SBOM, keypair, signature, and this proof that verification really detects tampering — into a verify-artifact job inside the inherited apply.yml, closing TM-02 from RISK-MAP.md.
Resources
- Sigstore — Signing Blobs — the same reference from lesson 6, with the verification section and its possible error results.
- NIST — Secure Hash Standard (FIPS 180-4) — the formal specification of SHA-256, the algorithm behind the avalanche effect this lesson demonstrated in Step 5.
- PKWARE — .ZIP File Format Specification — the ZIP format specification, the source for why the
End of Central Directoryindex is searched for from the end of the file, the exact technical reason behind Step 3.