Module 6: Supply Chain Sbom And Signing

6. Hands-on: signing and verifying the deployment artifact

Next →

Description

This is the lesson where TM-02 gets resolved for real: you're going to sign lambda/function.zip — the same 890-byte .zip terraform-and-iac-guide generated with data "archive_file" — with lesson 5's private key, and verify that signature with the public key, completely offline, with no public transparency log involved. This lesson also has an honest value worth announcing up front: the documentation this module's design is based on describes a set of cosign flags (--tlog-upload=false, --output-signature) that, when actually run against the version installed on this machine (v3.1.3), turn out to be deprecated — cosign changed its command-line interface between when that documentation was written and when this lesson was run. You're going to see the failed attempt, the real message explaining it, and the correct, current command, in that order — exactly as it happened while writing this guide.

Connection to the module

This is the module's most important lesson: manifest.sig, the file you produce here, is what lesson 7 is going to try to break on purpose, and what lesson 8 is going to verify as part of the pipeline. Everything you learned in lessons 4 and 5 — why a local keypair, how it was generated — converges here in the pair of commands that really matter: sign-blob and verify-blob.

Step 1 — The first attempt, exactly as the reference documentation describes it

The way to sign without uploading to the Rekor transparency log, according to Sigstore's official documentation consulted when designing this guide, uses the --tlog-upload=false flag when signing and --output-signature to save the signature to a file:

COSIGN_PASSWORD="" cosign sign-blob \
  --key cosign.key \
  --tlog-upload=false \
  --output-signature manifest.sig \
  lambda/function.zip

What to expect (literal, run to write this lesson):

Flag --tlog-upload has been deprecated, prefer using a --signing-config file with no transparency log services
Flag --output-signature has been deprecated, please use --bundle to provide the output bundle location, which will include the signature
Error: must specify --bundle with --new-bundle-format
error during command execution: must specify --bundle with --new-bundle-format

Read it with the same analytical attention you've already practiced with real failures from other tools in this guide: cosign doesn't fail because of a mistake on your part — it fails because its own command-line interface evolved since the source this module's design was based on was written. v3.1.3 introduced a new verification bundle format (Sigstore spec version v0.3) that replaces the --output-signature/--signature pair with a single --bundle file, which holds both the signature and all the verification material in one structured JSON document. This is exactly the hard rule that has governed this guide since its first lesson: if a command ran to write it, it ran for real — including the moment when the "correct according to the documentation" command turned out not to be correct anymore, and the current one had to be researched.

Step 2 — Why the correct flag alone isn't enough: the default Rekor exchange

Step 1's message suggests adding --bundle. Trying it, with nothing else, reveals a second, subtler and more important problem:

COSIGN_PASSWORD="" cosign sign-blob \
  --key cosign.key \
  --bundle manifest.sig \
  --yes \
  lambda/function.zip

This command does run without error — but not in the way this guide needs. Before showing why, it's worth stating precisely what it does by default: cosign v3.1.3, with --bundle and no other instruction, uses --use-signing-config=true (the default value), which queries a TUF-provided signing configuration with the URLs of Sigstore's public services — including Rekor, the transparency log —, and uploads the signature to that public log, exactly the keyless behavior lesson 4 explained and that this guide deliberately decided not to use. The resulting .sig/.bundle from that first attempt includes a complete tlogEntries block — with logIndex, rootHash, and a checkpoint signed by rekor.sigstore.dev — proof the upload really happened against the real public service.

   WHAT cosign sign-blob --bundle DOES BY DEFAULT (v3.1.3)
   ────────────────────────────────────────────────────────────

   [your command]  →  [queries TUF for the public signing config]  →  [signs]  →  [uploads to Rekor]
                                                                                          │
                                                                            requires a real network, and
                                                                            publishes the signature forever

   this is exactly the KEYLESS flow lesson 4 explained — and that this module does NOT use

This is the exact moment this guide had to stop and resolve the problem precisely, instead of accepting a result that contradicts its own design: signing like this isn't "almost offline" — it's a real keyless signature, with a real upload to a real public service, even though the command uses --key cosign.key. Lesson 4 already explained why that isn't what this module builds.

Step 3 — The real solution: a signing config explicitly without Rekor

cosign v3.1.3 exposes a mechanism for this — a signing config file you can generate empty, with no service declared, and pass to sign-blob so it doesn't query Fulcio, Rekor, or any timestamping service:

cosign signing-config create --out no-tlog-signing-config.json
cat no-tlog-signing-config.json

What to expect (literal, run to write this lesson):

{"mediaType":"application/vnd.dev.sigstore.signingconfig.v0.2+json", "rekorTlogConfig":{}, "tsaConfig":{}}

A minimal document, with no service URL declared at all — no fulcioCertificateAuthorityUrls, no rekorTlogUrls, no tsaUrls. With this file passed explicitly via --signing-config, cosign has no service to query or upload anything to, and the signing command is, now, truly completely local:

COSIGN_PASSWORD="" cosign sign-blob \
  --key cosign.key \
  --signing-config no-tlog-signing-config.json \
  --bundle manifest.sig \
  --yes \
  lambda/function.zip

What to expect (literal, run to write this lesson):

Using payload from: lambda/function.zip
Signing artifact...
Wrote bundle to file manifest.sig

No warning at all about personal data or public transparency — the legal consent cosign asks for before uploading something to a hosted service, which did not appear in this run, because there's no upload to authorize. Compare it, if you want to confirm it with your own eyes, with what you would have seen in Step 2: that command does show, before signing, a complete legal notice about Sigstore's immutable public log, which you have to accept by typing y (or passing --yes, as in Step 2). This step's complete absence of that notice is, by itself, confirmation that no hosted service is involved.

Step 4 — Confirming manifest.sig contains no Rekor record

cat manifest.sig

What to expect (the mediaType field and the structure are literal; the signature field is variable — see the note below):

{
  "mediaType": "application/vnd.dev.sigstore.bundle.v0.3+json",
  "verificationMaterial": {
    "publicKey": {
      "hint": "rcWoHarzQrHVD4Fsb2wPlD9/X+ZVuFA1F2yWL+A2EQk="
    }
  },
  "messageSignature": {
    "messageDigest": {
      "algorithm": "SHA2_256",
      "digest": "mX7XF7LjI00UJ+DFC6+/IXEoh/Cdsb9EOPAWXwmwn1U="
    },
    "signature": "MEUCIQDD82ke5ovVwabmMe+2xV1S1XMMwtiRA5veNQn7cb2PFQIgM5Dxo8B8358GSie2lKtX9kMS4lRx4NDd9mm5f1DmzSw="
  }
}

Three observations, in order of importance:

  • There's no tlogEntries field at all — unlike Step 2's failed attempt, this bundle only contains what's strictly necessary to verify the signature with a local key: the public key's verification material (verificationMaterial.publicKey) and the signature itself (messageSignature). Nothing from Rekor, no public checkpoint. Confirm it yourself:

    grep -c tlogEntries manifest.sig

    What to expect (literal): 0 — the pattern doesn't show up even once in the entire file.

  • messageDigest.digest is literal, and matches exactly the hash you already know. mX7XF7LjI00UJ+DFC6+/IXEoh/Cdsb9EOPAWXwmwn1U= is the same Base64-encoded SHA-256 value terraform-and-iac-guide (Module 7, lesson 3) reported for function.zip with output_base64sha256, and that you independently confirmed with shasum -a 256. It isn't a coincidence: it's proof you're signing exactly the same artifact, byte for byte, that this ecosystem's three previous guides built — cosign calculated its own hash of the file, and that hash is identical to the one Terraform calculated on its own, months of content earlier, with a completely different tool.

  • messageSignature.signature is VARIABLE — never expect this exact value to repeat. ECDSA (cosign generate-key-pair's default algorithm) is, by design, non-deterministic: every signature over the same content, with the same key, produces different bytes, because the algorithm incorporates a random value in every signing operation (this is a deliberate security property of the ECDSA scheme, not a flaw). If you regenerate manifest.sig ten times over the same function.zip without changing anything, you're going to get ten different signature values, and all ten are going to verify correctly against cosign.pub — exactly what the next lesson confirms with the only property that actually matters: not the signature's exact value, but the result of verifying it.

Step 5 — Verifying the signature, 100% offline

cosign verify-blob \
  --key cosign.pub \
  --bundle manifest.sig \
  --insecure-ignore-tlog=true \
  lambda/function.zip

What to expect (literal, run to write this lesson):

WARNING: Skipping tlog verification is an insecure practice that lacks transparency and auditability verification for the blob.
Verified OK

Two lines, and both matter. The first is an honest warning from cosign itself: without a public transparency log involved, nobody but you can independently confirm this signature existed at the moment it claims to — it's exactly the trade-off lesson 4 explained when choosing keypair over keyless, and cosign reminds you of it every time, instead of leaving it implicit. --insecure-ignore-tlog=true is the flag that explicitly acknowledges that warning: it tells cosign "I know there's no Rekor involved, verify with just the public key, as befits this mode." The second line, Verified OK, is the result that matters: the signature in manifest.sig mathematically corresponds to lambda/function.zip signed with the private key that corresponds to cosign.pub.

echo $?
0

Exit code 0 — the one any pipeline job (lesson 8 is going to use it exactly this way) interprets as "continue"; a non-zero value, as you'll see in lesson 7, stops the chain right there.

What you just proved, precisely

It's worth stating precisely what this chain of commands guarantees, without exaggerating or downplaying it: Verified OK confirms that the lambda/function.zip file, as it exists on disk at the moment you run verify-blob, is exactly the same file, byte for byte, that existed at the moment you ran sign-blob in Step 3 — not one byte added, not one removed, not one modified —, and that whoever signed it had access to the private key corresponding to cosign.pub. It doesn't confirm the code inside that .zip is free of bugs, nor that it's "secure" in the sense of conftest's policies (Module 4) or Trivy/Checkov's scans (Module 5) — those are different questions, answered by different tools. What this signature answers is a single question, with mathematical precision: is this, really, the artifact someone with the private key approved, with no alteration along the way?

Common mistakes

Copying the --tlog-upload=false/--output-signature flags from a documentation source without checking them against the installed version (this lesson's central mistake, left in on purpose). What happens: someone finds those flags in an article, a tutorial, or even an older cosign version's documentation, and uses them as-is against a more recent install. How to spot it: the Flag ... has been deprecated message appears in the output, unambiguously. How to fix it: exactly this lesson's Step 1 — read the complete deprecation message, not just the final error; it almost always names the correct replacement (--bundle in this case). cosign --help (or cosign sign-blob --help) against the real installed version is always a more reliable source of truth than any external documentation, because it describes exactly the behavior of the version you have on your machine right now.

Using --bundle without --signing-config, and accidentally uploading a real signature to Rekor's public log (configuration mistake, the most serious one in this lesson because of its irreversible effect). What happens: someone resolves Step 1's error by adding --bundle, sees the command run without failing, and assumes the problem is resolved — without noticing that, by default, cosign v3.1.3 still queries Sigstore's public signing configuration and uploads the signature to Rekor, a public, immutable log. How to spot it: check the resulting .bundle file with grep -c tlogEntries file.bundle — if the count is greater than zero, the signature really did get uploaded, and there's no way to undo it: Rekor is, by design, a permanent log. How to fix it: always pass --signing-config with a file declaring no services (this lesson's Step 3) when the goal is a completely local signature — never assume the absence of an error message means the behavior was the expected one.

Expecting two sign-blob runs over the same file to produce the same signature, and suspecting an error if they don't match (ECDSA expectation mistake). What happens: someone signs function.zip twice, compares the manifest.sig files byte for byte, sees they're different, and concludes something failed. How to spot it: if your "consistency" test is comparing messageSignature.signature's exact value between runs. How to fix it: as Step 4 explained, ECDSA is non-deterministic by design — the exact signature will vary on every run, and that's correct, not an error. The correct consistency test is never comparing signature bytes; it's running cosign verify-blob on each and confirming both return Verified OK.

Exercises

Exercise 1 — Explain, to someone who only saw this lesson's final result, why Step 1 "failed" and that was correct, not a mistake in the guide. A colleague, seeing the Error: must specify --bundle message, asks why a published guide would include a command that fails. How would you justify it?

See solution

A complete answer appeals to this guide's hard rule: every command that appears really ran to write the lesson, including the ones that didn't work the way the reference documentation suggested. Showing Step 1 as-is — with its real error — is more honest and more useful than "silently fixing" the final command without explaining why it changed: someone who investigates cosign sign-blob on their own, using an outdated source, is going to run into exactly this same error, and this lesson gives them both the diagnosis (cosign changed its interface) and the solution (--bundle plus --signing-config), instead of leaving them lost in front of a message they don't understand.

Exercise 2 — Predict what grep -c tlogEntries would show over the .bundle you would have gotten in Step 2 (the attempt without --signing-config), and explain why that number differs from Step 4's. Without running Step 2's command again, what value would you expect, and what does the difference mean?

See solution

Step 2 (without --signing-config, with --use-signing-config=true implicit) would produce a .bundle with tlogEntries present at least once — in fact, as a large nested JSON block, with logIndex, logId, inclusionProof, and a checkpoint signed by rekor.sigstore.dev —, while Step 4 (with --signing-config no-tlog-signing-config.json) gives 0. The difference isn't cosmetic: in Step 2, the signature really did get uploaded to a real public service and stays there permanently; in Step 4, the signature never left your machine. It's the practical, verifiable difference between the keyless mode lesson 4 explained (though here triggered by accident, by omitting a flag) and the completely local keypair mode this module deliberately builds.

Exercise 3 — Decide what you'd tell a colleague who proposes simplifying the flow by removing --signing-config "since it works the same anyway." Someone on your team argues that, since Step 2 also produces Verified OK in the end, the extra hassle of generating and passing no-tlog-signing-config.json isn't worth it. Do you agree?

See solution

No, and the reason isn't about convenience but about real consequence: even though both flows end in a verifiable signature, Step 2 publishes the signature — and, with it, the artifact's hash and associated metadata — to a permanent public log, without anyone explicitly deciding to do that. For an 890-byte practice artifact in a $0 lab, the harm is minimal, but the habit being formed isn't: in a real project, accidentally uploading an artifact (or its hash) to an immutable public log could leak information the organization didn't want made public, with no way to undo it afterward. The extra cost of generating no-tlog-signing-config.json once — a one-line file, reusable for every future signature — is trivial compared to the irreversibility of an accidental upload to Rekor.

Summary and next step

In this lesson you signed lambda/function.zip for real, with cosign sign-blob, and verified that signature with cosign verify-blob, getting Verified OK completely offline. Along the way, you precisely documented a real case of drift between reference documentation and the tool's current behavior — the --tlog-upload=false/--output-signature flags turned out to be deprecated in v3.1.3, and the correct path requires --bundle combined with an explicitly empty --signing-config to reliably avoid a real upload to Rekor's public log. You confirmed that the artifact's hash cosign calculated matches, byte for byte, the one terraform-and-iac-guide reported, and learned to distinguish which parts of a signing bundle are literal (the messageDigest) from those that vary by design on every run (the signature itself).

Before moving on you should be able to: explain, without looking at this lesson, why --bundle alone isn't enough for a completely offline signature; read a cosign flag's deprecation message and find its correct replacement without relying on an external source; and justify why comparing exact signature bytes is never the correct way to verify consistency between two sign-blob runs.

Lesson 7 really puts this signature to the test: you're going to modify function.zip after signing it, and confirm, live, that cosign verify-blob detects it and fails explicitly.

Resources

  1. Sigstore — Signing Blobs — the official sign-blob/verify-blob reference, including the v0.3 bundle format this lesson uses.
  2. GitHub — sigstore/cosign issue #4503 — the public discussion about --tlog-upload's behavior and its replacement, the same drift this lesson documented firsthand.
  3. Sigstore — Bundle specification — the formal specification of the application/vnd.dev.sigstore.bundle.v0.3+json format manifest.sig produces.
  4. terraform-and-iac-guide, Module 7, lesson 3 (03-packaging-lambda-code-with-archive-file.md) — the exact origin of the hash mX7XF7LjI00UJ+DFC6+/IXEoh/Cdsb9EOPAWXwmwn1U= this lesson independently confirmed, with a completely different tool.
← Previous5. Hands-on: installing `cosign` and a local keypairNext →7. Hands-on: breaking the chain on purpose