5. From HCL to JSON: the `terraform plan` as `conftest`'s input

Description

Everything lesson 4 did against an eleven-line YAML, this lesson does against andes-cargo-infra/'s complete terraform plan — the same project terraform-and-iac-guide and this guide's Module 2 already built. conftest can't read HCL directly; it needs the plan converted to JSON, with terraform show -json. This lesson runs that command for real, and explores the resulting structure until you know, without hesitation, where every piece the lessons 6 and 7 policies will read lives.

Connection to the module

This is the entire module's hinge lesson: before this point, every example was deliberately trivial (a test YAML); after this point, every policy runs against Andes Cargo's real, complete input. Nothing in lessons 6, 7, and 8 makes sense without having explored, with your own eyes, this JSON's exact shape.

Analogy: the complete X-ray, not the patient's photo

A photo of a patient tells you what they look like from outside — useful, but limited: you can't see a fracture under the skin with a photo. An X-ray is a different representation of the same patient, designed specifically so a machine — or a trained eye — can detect structures a photo would never show. andes-cargo-infra/'s HCL is the photo: readable, expressive, meant for a person to write and read. The terraform plan in JSON is the X-ray: a different representation of the same project, designed specifically so a machine — conftest, in this case — can examine every proposed change with structural precision, field by field, with no ambiguity of interpretation.

Step 1 — terraform plan -out=tfplan: the plan saved to disk

You already know terraform plan from terraform-and-iac-guide — what's new here is the -out flag, which saves the plan's result to a binary file, instead of only printing it to the terminal:

cd andes-cargo-infra
terraform init
terraform plan -out=tfplan

What to expect (literal, executed to write this lesson — the init):

Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 6.0"...
- Finding hashicorp/archive versions matching "~> 2.0"...
- Installing hashicorp/aws v6.60.0...
- Installed hashicorp/aws v6.60.0 (signed by HashiCorp)
- Installing hashicorp/archive v2.8.0...
- Installed hashicorp/archive v2.8.0 (signed by HashiCorp)

Terraform has been successfully initialized!

What to expect (literal — the plan, header and closing; the full body is the resources you already know from previous modules):

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create
 <= read (data resources)

Terraform will perform the following actions:

  # data.aws_iam_policy_document.lambda_dynamodb_write will be read during apply
  # (config refers to values not yet known)
 <= data "aws_iam_policy_document" "lambda_dynamodb_write" {
      + id            = (known after apply)
[...]

Plan: 14 to add, 0 to change, 0 to destroy.

Saved the plan to: tfplan

Plan: 14 to add — twelve business resources inherited from terraform-and-iac-guide plus the two this guide's Module 2 added (aws_iam_openid_connect_provider and modules/oidc-provider/'s aws_iam_role). tfplan, the file that just appeared on your disk, isn't readable text — it's Terraform's internal binary format, optimized so terraform apply tfplan can execute it exactly as planned, without recalculating anything. Neither conftest nor any program outside Terraform can read this file directly — that's what Step 2 is for.

Step 2 — terraform show -json: the conversion to JSON

terraform show -json tfplan > tfplan.json

This command recalculates nothing — it takes Step 1's binary tfplan and serializes it completely to JSON, without losing a single field. The result is a single file, potentially hundreds of kilobytes on a large project, with the plan's exact structure in a format any language with JSON support — Rego included — can read unambiguously.

What to expect (confirming the file exists and has real content):

ls -la tfplan.json

-rw-r--r--  1  user  staff  38214  tfplan.json

(The exact size in bytes varies by project and provider version — what is literal and deterministic is the structure you explore next, not the file's weight.)

Step 3 — Exploring the structure: the top-level keys

jq 'keys' tfplan.json

What to expect (literal, executed to write this lesson):

[
  "applyable",
  "checks",
  "complete",
  "configuration",
  "errored",
  "format_version",
  "output_changes",
  "planned_values",
  "prior_state",
  "relevant_attributes",
  "resource_changes",
  "terraform_version",
  "timestamp",
  "variables"
]

Fourteen keys in total, but only three matter for everything this module does: format_version and terraform_version (context metadata, useful for confirming which Terraform version a plan was generated against), and resource_changes — the complete list of every resource this plan is going to create, modify, destroy, or simply read. Every policy you write in lessons 6 and 7 iterates over input.resource_changes; the rest of the fourteen keys fall outside this module's scope.

jq '{format_version, terraform_version}' tfplan.json

What to expect (literal):

{
  "format_version": "1.2",
  "terraform_version": "1.15.8"
}

Step 4 — resource_changes: the list policies iterate over

jq '.resource_changes | length' tfplan.json

What to expect (literal):

16

Sixteen, not fourteen — and the difference is worth understanding before moving on. Plan: 14 to add counts only managed resources (aws_dynamodb_table, aws_iam_role, and the rest) Terraform is going to create. resource_changes also includes the data sources — two, in this project — whose action isn't create but read: they aren't infrastructure being created, they're queries Terraform resolves in order to calculate the rest of the plan. Confirm it yourself:

jq -r '.resource_changes[] | "\(.address) | \(.type) | \(.change.actions)"' tfplan.json

What to expect (literal, executed to write this lesson):

data.aws_iam_policy_document.lambda_dynamodb_write | aws_iam_policy_document | ["read"]
aws_dynamodb_table.shipments | aws_dynamodb_table | ["create"]
aws_iam_role_policy.lambda_write_shipments | aws_iam_role_policy | ["create"]
aws_lambda_function.process_shipment_manifest | aws_lambda_function | ["create"]
aws_lambda_permission.allow_s3_invoke | aws_lambda_permission | ["create"]
aws_s3_bucket_notification.shipment_docs_trigger | aws_s3_bucket_notification | ["create"]
module.app_server_role.aws_iam_role.this | aws_iam_role | ["create"]
module.app_server_role.aws_iam_role_policy.this | aws_iam_role_policy | ["create"]
module.github_oidc.data.aws_iam_policy_document.trust | aws_iam_policy_document | ["read"]
module.github_oidc.aws_iam_openid_connect_provider.github_actions | aws_iam_openid_connect_provider | ["create"]
module.github_oidc.aws_iam_role.deploy | aws_iam_role | ["create"]
module.lambda_manifest_processor_role.aws_iam_role.this | aws_iam_role | ["create"]
module.lambda_manifest_processor_role.aws_iam_role_policy.this | aws_iam_role_policy | ["create"]
module.shipment_docs_bucket.aws_s3_bucket.this | aws_s3_bucket | ["create"]
module.shipment_docs_bucket.aws_s3_bucket_policy.this[0] | aws_s3_bucket_policy | ["create"]
module.shipment_docs_bucket.aws_s3_bucket_versioning.this[0] | aws_s3_bucket_versioning | ["create"]

Two rows with ["read"] (the two data "aws_iam_policy_document" blocks whose value depends on an ARN not yet calculated — you'll see this again in lesson 7, when one of these two affects what a policy can or can't verify), fourteen rows with ["create"]. .change.actions is always a list, never a single value — because a real Terraform action can be compound: ["delete", "create"] describes a full replacement (destroy and recreate, the pattern you already saw in terraform-and-iac-guide with conditional count), not two independent actions. Any policy in this module that looks for destructions — lesson 6, specifically — has to test whether "delete" is contained in this list, not whether the list is exactly ["delete"].

Step 5 — A complete resource_change, field by field

jq '.resource_changes[] | select(.address == "aws_dynamodb_table.shipments")' tfplan.json

What to expect (literal, trimmed to the fields this module uses; the actual full object includes more Terraform internal-state metadata, irrelevant to this module's policies):

{
  "address": "aws_dynamodb_table.shipments",
  "type": "aws_dynamodb_table",
  "change": {
    "actions": [
      "create"
    ],
    "before": null,
    "after": {
      "name": "Shipments",
      "hash_key": "shipmentId",
      "billing_mode": "PAY_PER_REQUEST",
      "tags": {
        "Environment": "dev",
        "ManagedBy": "terraform",
        "Project": "andes-cargo"
      }
    }
  }
}

Four fields you're going to use in every policy for the rest of this module:

• address — the resource's unique identifier within the plan, identical to what you'd use with terraform state show. Lesson 6 uses it to point, in the deny message, at exactly which resource violated the rule.
• type — the Terraform resource type (aws_dynamodb_table, aws_iam_role_policy, aws_s3_bucket...). Every policy in this module filters resource_changes by this field before looking at anything else — lesson 6 only cares about aws_dynamodb_table, lesson 7 only about aws_iam_role_policy and aws_s3_bucket/aws_s3_bucket_public_access_block.
• change.before — the resource's state before this plan. For a resource being created for the first time (like this one), before is null — it didn't exist. For a resource being destroyed, before has the resource's complete content as it existed; after is null.
• change.after — the proposed state, after applying the plan. This is the field most policies in this module inspect: what hash_key the table has, what Action each Statement of an IAM policy has, what value each aws_s3_bucket_public_access_block flag has.

Going deeper: before/after, and why some values are missing

A real detail you'll run into again in lesson 7, so it's worth seeing here first, in a simpler case: not every field of after is always present. When an attribute's value depends on something Terraform still can't calculate at plan time — typically, a reference to another resource's ARN or id that doesn't exist yet — that field simply doesn't appear in after; instead, it appears marked as known-after-apply in a separate section of the JSON (after_unknown), outside this module's scope. A Rego policy that assumes a field is always present can fail silently — not with an error, but with a condition that simply never holds, because it tried to read a key that doesn't exist — if it doesn't account for this case. You'll see the concrete, real example of this in lesson 7: the least-privilege policy needs to read each aws_iam_role_policy's policy field, and one of Andes Cargo's three real roles has that field absent in this specific plan, for this exact reason.

Common mistakes

Running terraform show -json before terraform plan -out=, over a file that doesn't exist yet (ordering mistake). What happens: someone, in a hurry, runs terraform show -json tfplan > tfplan.json without having generated tfplan first. How to spot it: the command fails with an explicit error, something like Error: Failed to read the given file as a state or plan file — it doesn't produce an empty or corrupt tfplan.json, it stops immediately. How to fix it: the order is always plan -out= first, show -json after — the second command reads the binary file the first one produced, it can't generate anything on its own.

Confusing Plan: N to add with the number of elements in resource_changes (counting mistake, this lesson's specific error). What happens: someone sees Plan: 14 to add in the text output, then counts resource_changes with jq and gets 16, and concludes something's wrong or the JSON has a bug. How to spot it: if your first reaction to the two different numbers is "this doesn't add up." How to fix it: there's no error — Plan: N to add counts only managed resources with create/update/delete actions; resource_changes also includes data sources with read actions, which never show up in the text summary because they aren't infrastructure Terraform is going to create or change. This lesson confirmed it with the same project: 14 managed resources, 2 data reads, 16 total.

Writing a policy that assumes change.actions always has a single element (data-structure mistake). What happens: someone writes rc.change.actions == "delete" (a direct comparison against a string) instead of "delete" in rc.change.actions (a list-membership check). How to spot it: the policy never fires, not even against a plan that does destroy the resource — because change.actions is ["delete"], a one-element list, and a list is never equal to a string even if it contains that single string. How to fix it: always use Rego's in operator against change.actions, exactly as you'll see in lesson 6's policy — never a direct equality comparison, because a replacement action (["delete", "create"]) has more than one element.

Exercises

Exercise 1 — Count how many resources in this plan are of type aws_iam_role_policy, using jq. Without looking at this lesson's complete listing, write the jq command that filters resource_changes by type == "aws_iam_role_policy" and counts the result. How many would you expect to find, based on what you know from this guide's Module 2?

jq '[.resource_changes[] | select(.type == "aws_iam_role_policy")] | length' tfplan.json

Exercise 2 — Explain, to a colleague, why resource_changes has entries with actions: ["read"]. Your colleague asks: "Why are there data sources in a file that's supposed to describe what's going to get created?" Answer in two or three sentences.

Exercise 3 — Predict what would happen to change.before and change.after if this same plan instead described the Shipments table's destruction. Without having read lesson 6 yet, predict: for a resource being destroyed (not created), what would you expect to find in change.before and change.after? Justify using what you already know from this lesson's Step 5.

Summary and next step

In this lesson you converted andes-cargo-infra/'s terraform plan to JSON with terraform show -json, and explored its real structure with jq: fourteen top-level keys, of which only resource_changes matters for this module; sixteen entries in that list — fourteen managed resources with create, two data sources with read; and a resource_change's complete anatomy — address, type, change.before, change.after — that every policy in the next two lessons is going to read.

Before moving on you should be able to: generate tfplan.json from scratch, with this lesson's exact two commands; explain the difference between Plan: N to add and resource_changes's total count; and predict which fields (before/after) a resource would have depending on whether it's being created, modified, or destroyed.

Lesson 6 uses, for the first time, input.resource_changes inside a real policy: no-destroy-shipments.rego, the one that replaces cicd-and-gitops-on-aws-guide's handcrafted grep — and the one you're going to watch really fail, against a plan that does destroy the table.

Resources

1. Terraform CLI — terraform show — the official reference for the command that produces tfplan.json, including the -json flag.
2. Terraform — JSON Output Format — the complete specification for a plan's JSON format, including resource_changes, change.actions, and after_unknown.
3. jq — Official manual — the tool used in this lesson to explore the JSON before writing Rego against the same structure.
4. This module, lesson 1 — the confirmation that neither this lesson nor any other in this module needs LocalStack: terraform plan computes a local diff, touching no remote resource.