The metadata file: schema, partitioning, snapshots

Description

This lesson opens the file metadata_location points to — the case file's cover sheet, in this module's analogy: kiosko.fact_orders's metadata file, with a .metadata.json extension. Unlike the manifest files and manifest lists (lesson 4), this file is plain JSON, readable with any text editor or with Python's json.load() with no extra dependency at all. You're going to read it directly, field by field, and you're going to recognize three things you already know in it: the exact schema you declared in module 1's lesson 5, the absence of partitioning (still), and a single snapshot, the one table.append() created in that module's lesson 6.

Connection to the module. Lesson 2 confirmed the catalog only stores an address. This lesson opens what's at that address — the file's complete cover sheet, with all the information the catalog, on purpose, doesn't store directly.

An analogy: reading the complete cover sheet

A court case file's cover sheet, unlike the desk that only gave you an address, does contain substantial information: the case number, the date of the most recent hearing, a summary of what was decided at each previous hearing, and an exact reference to where the current hearing's evidence index is. It doesn't contain the evidence itself — that's one level further in — but it does contain everything needed to know what shape the case has: its parties, its hearing history, its current status. That is, precisely, what an Iceberg metadata file contains: the table's complete schema (which columns exist, with which types), the partition scheme (how the data is organized, if it is at all), and the complete list of snapshots — every version of the table that ever existed, with its date and its summary.

Worked example: opening the complete cover sheet

Step 1 — Find the exact path via the catalog, and open it with json.load()

# inspect_metadata_json.py
import json
import os
import sqlite3
from urllib.parse import urlparse

catalog_db_path = os.path.abspath("kiosko_catalog.db")
conn = sqlite3.connect(catalog_db_path)
cur = conn.cursor()
cur.execute("SELECT metadata_location FROM iceberg_tables WHERE table_name = 'fact_orders';")
metadata_location = cur.fetchone()[0]
conn.close()

# metadata_location is a file:// URI; urlparse turns it into a normal system path
metadata_path = urlparse(metadata_location).path
with open(metadata_path) as f:
    metadata = json.load(f)

print("File opened directly with json.load() -- it's plain, readable JSON:")
print(" ", os.path.basename(metadata_path))

What to expect:

File opened directly with json.load() -- it's plain, readable JSON:
  00001-<uuid>.metadata.json

Not a single Iceberg library involved — Python's standard library json.load() opens the file with no error at all, exactly the way you'd open any .json file. That, in practice, is the central difference that sets this file apart from the manifests in lesson 4.

Step 2 — The main pointers: version, current schema, current snapshot

print("=== format-version and main pointers ===")
print("format-version:", metadata["format-version"])
print("table-uuid:", metadata["table-uuid"], " <- unique per table, not per run")
print("location:", metadata["location"])
print("current-schema-id:", metadata["current-schema-id"])
print("current-snapshot-id:", metadata["current-snapshot-id"])

What to expect (table-uuid and current-snapshot-id are values assigned in your own run — different from what's shown here; everything else is identical on any run that exactly reproduces module 1):

=== format-version and main pointers ===
format-version: 2
table-uuid: <uuid assigned to your table, different from any other run>
location: file:///.../kiosko_warehouse/kiosko/fact_orders
current-schema-id: 0
current-snapshot-id: <snapshot-id assigned in your run, different each time>

format-version: 2 confirms this table uses version 2 of the Iceberg specification — the current one as this guide is being written, with full support for row-level deletes and a sequence-number per snapshot. table-uuid is a unique identifier, generated once when catalog.create_table() created the table in module 1's lesson 5 — it never changes, not even between snapshots; be careful to tell it apart from current-snapshot-id, which does change with every write.

Step 3 — The schema, field by field

print("\n=== schemas[] -- the schema declared in M1 lesson 5 ===")
for field in metadata["schemas"][0]["fields"]:
    print(f"  id={field['id']:<2} {field['name']:<12} {field['type']:<10} required={field['required']}")

What to expect:

=== schemas[] -- the schema declared in M1 lesson 5 ===
  id=1  order_id     string     required=True
  id=2  store_id     string     required=True
  id=3  product_id   string     required=True
  id=4  quantity     int        required=True
  id=5  unit_price   double     required=True
  id=6  revenue      double     required=True
  id=7  order_ts     timestamp  required=True

You recognize these seven lines: they're, field by field, the Schema you declared with NestedField in module 1's lesson 5 — same field_ids (id here), same names, same types, same requiredness. The metadata file doesn't reinterpret the schema — it stores it as is, as the single source of truth any reader must consult before opening a single Parquet file.

Step 4 — Partitioning: still empty

print("\n=== partition-specs[] -- no partitioning (M1-M4), M5 evolves this ===")
print(metadata["partition-specs"])

What to expect:

=== partition-specs[] -- no partitioning (M1-M4), M5 evolves this ===
[{'spec-id': 0, 'fields': []}]

A PartitionSpec with spec-id: 0 and an empty fields list — kiosko.fact_orders isn't partitioned, neither by folders nor in any other way, exactly as you created it in module 1's lesson 5. Module 5 of this guide introduces the first real partition scheme, over a different table (kiosko.fact_orders_at_scale); this table stays unpartitioned for the whole guide.

Step 5 — The list of snapshots, and the main reference

print("\n=== snapshots[] -- ONE single snapshot after the M1 load ===")
print("number of snapshots:", len(metadata["snapshots"]))
snap = metadata["snapshots"][0]
print("  snapshot-id:", snap["snapshot-id"], " <- not reproducible, different on your run")
print("  sequence-number:", snap["sequence-number"])
print("  operation:", snap["summary"]["operation"])
print("  manifest-list:", os.path.basename(urlparse(snap["manifest-list"]).path))
print("  summary.added-data-files:", snap["summary"]["added-data-files"])
print("  summary.added-records:", snap["summary"]["added-records"])
print("  summary.total-records:", snap["summary"]["total-records"])

print("\n=== refs -- the 'main' branch points to the current snapshot ===")
print(metadata["refs"])

What to expect (snapshot-id isn't reproducible; the rest of the structure and the business values are):

=== snapshots[] -- ONE single snapshot after the M1 load ===
number of snapshots: 1
  snapshot-id: <snapshot-id assigned in your run, different each time>
  sequence-number: 1
  operation: append
  manifest-list: snap-<snapshot-id>-0-<uuid>.avro
  summary.added-data-files: 1
  summary.added-records: 40
  summary.total-records: 40

=== refs -- the 'main' branch points to the current snapshot ===
{'main': {'snapshot-id': <snapshot-id assigned in your run, different each time>, 'type': 'branch'}}

This is the heart of the lesson: metadata["snapshots"] has exactly one entry, with sequence-number: 1 — the first commit that ever existed on this table — operation: "append" — the same operation you ran in module 1's lesson 6 — and summary.added-records: 40/total-records: 40 — the same count you already verified. And notice the manifest-list field: it's the exact reference to the next link in the chain, the one lesson 4 opens. refs["main"] confirms, once again, which snapshot is current — the same current-snapshot-id from step 2 — using the "branch" mechanism Iceberg has supported since version 2 of its spec, even though this guide never uses a branch other than main.

Diagram: what lives inside the metadata file

flowchart TB
    META["metadata.json"]
    META --> A["format-version, table-uuid, location"]
    META --> B["schemas[] + current-schema-id\n(the complete Schema, with field_id)"]
    META --> C["partition-specs[] + default-spec-id\n(empty: no partitioning yet)"]
    META --> D["snapshots[] + current-snapshot-id\n(1 snapshot: append, 40 records)"]
    META --> E["refs.main -> current snapshot"]
    D -->|"snapshot.manifest-list points to"| NEXT["Next link:\nmanifest list (lesson 4)"]

Going deeper: why snapshots[] is a list, not a single object

It's worth noting something this lesson doesn't exploit yet, but that all of module 3 is going to turn into this guide's central guarantee: metadata["snapshots"] is a list, not a single object — Iceberg's format is designed, from its spec, to accumulate more than one snapshot in the same metadata file. At this point in the guide it has exactly one element because there was only one write, but if in module 3 you do a second overwrite() on another table, you're going to see that list grow to two elements, with current-snapshot-id always pointing to the most recent one — without the first one disappearing. This is, precisely, the data structure that makes time travel possible: it isn't magic, it's a list of snapshots Iceberg never prunes on its own, with a pointer (current-snapshot-id) that says which one is "current now" — exactly the same as the catalog (lesson 2) says which metadata file is "current now" — the same pattern, repeated at two different levels of the chain.

Common mistakes

Looking for the actual row values (order_id, revenue, etc.) inside the metadata file. What happens: someone, seeing how much information this file contains, expects to find, right there in some field, Kiosko's forty orders. Why it happens: the metadata file is surprisingly rich in detail — schema, history, summaries — so it's easy to overestimate how much it contains. How to spot it: if your search for "ORD-1001" (or any real order_id) inside the JSON finds nothing, confirm you're in the right file — the absence is expected, not an error. How to fix it: the metadata file contains summaries (summary.added-records: 40), never the rows themselves — those live, several links further down the chain, in the Parquet files (lesson 5).

Confusing current-schema-id with field_id. What happens: someone sees "current-schema-id": 0 and, separately, "id": 1 inside each schema field, and assumes they're the same kind of number with interchangeable roles. Why it happens: both are small integers, and "schema" and "field" sound close together. How to spot it: if your code tries to use current-schema-id to identify a specific column, revisit what each one represents in this lesson's worked example. How to fix it: current-schema-id identifies a complete version of the schema (useful once module 4 evolves the schema and a schema-id: 1 appears); the id inside each field (field_id, in module 1's lesson 5 terminology) identifies an individual column, stable even as the schema evolves. They're two different levels of identification, each with its own purpose.

Editing the metadata file by hand "to quickly fix something." What happens: someone, seeing the file is readable, editable JSON, opens it in a text editor and modifies a value directly, thinking it's a valid way to fix an error. Why it happens: the file's readability deceptively invites treating it like any other config file. How to spot it: if after editing the file by hand, catalog.load_table("kiosko.fact_orders") starts failing, or the table behaves inconsistently with what the catalog expects, check whether the file is still exactly what Iceberg wrote. How to fix it: never edit a metadata file by hand — any real change to an Iceberg table must go through an API operation (table.append(), table.overwrite(), table.update_schema(), etc.), which is the only way for the resulting new metadata file to be consistent with the manifests and the data it points to.

Exercises

Exercise 1 — Reproduce the five parts yourself. On your own machine, with module 1's state available, run the five parts of this lesson's worked example. Confirm you see exactly one entry in snapshots[], with operation: "append" and summary.added-records: 40.

See solution

If your table has the same single snapshot module 1 left, your output should structurally match what's shown here in everything except table-uuid, current-snapshot-id, the snapshot-id inside snapshots[0], and the exact name of the manifest list file — all of those are values your own run assigned, different from any other run. If len(metadata["snapshots"]) isn't 1, check whether you ran table.append() more than once in module 1's lesson 6.

Exercise 2 — Locate the snapshot's manifest-list, and write down its name. Using this lesson's script, extract the value of snap["manifest-list"] and write down only the file name (without the full path). What pattern do you recognize in that name, compared to the snapshot-id you already wrote down?

See solution

The file name follows the pattern snap-<snapshot-id>-0-<uuid>.avro — the snapshot-id itself (the same large integer you saw in current-snapshot-id) shows up, literally, as part of the manifest list file's name. This naming pattern isn't a coincidence: it means that just by looking at the name of a snap-*.avro file in the metadata/ folder, you immediately know which snapshot it belongs to, without having to open it — something lesson 6 of this module takes advantage of when walking the full directory from the terminal.

Exercise 3 — Prediction: what would change in this file if module 3 did a second overwrite()? Without jumping ahead to module 3 yet, predict: if later in this guide a second write ran against a table (an overwrite(), for example), which fields of this same metadata file do you expect to change, and which do you expect to stay exactly the same?

See solution

current-snapshot-id would change, pointing to the new snapshot; snapshots[] would grow to two elements, with the old snapshot still present in the list (not replaced); refs["main"]["snapshot-id"] would also change, along with current-snapshot-id. On the other hand, table-uuid would stay exactly the same — it's the table's permanent identifier, not a version's — and schemas[]/current-schema-id would also stay the same, unless the write also included a schema evolution (module 4), which is a different operation. This lesson's Going deeper section already previewed exactly this idea: snapshots[] is designed to grow, never to replace its earlier content.

Summary and next step

In this lesson you opened kiosko.fact_orders's metadata file directly with json.load() — with no Iceberg dependency at all — and confirmed, field by field, that it contains the complete schema (seven columns, same field_ids from module 1's lesson 5), an empty partition-specs (no partitioning yet), and a list of snapshots with exactly one entry: the one table.append() created in that module's lesson 6, with 40 records added.

Before moving on you should be able to: name a metadata file's four main sections (schemas, partition-specs, snapshots, refs); and explain why snapshots[] is a list that grows, never a single object that gets overwritten.

Lesson 4 follows the arrow once more: it opens the manifest-list field of the snapshot you just read — and there, direct reading with json.load() stops working, because the next link is no longer JSON.

Resources

• Apache Iceberg — official documentation, "Table Spec," the table metadata section (TableMetadata), the formal definition of every field this lesson read. iceberg.apache.org/spec. In English.
• PyIceberg — API reference, Table.metadata and the TableMetadata class, the typed representation of the same file this lesson read as raw JSON. py.iceberg.apache.org/api. In English.
• This guide's DESIGN doc — the metadata file's exact naming format and the warning about never hardcoding a snapshot-id, applied again in this lesson. src/guides/lakehouse-and-iceberg-guide/DISENO.md. In Spanish.