The catalog: a pointer to the current metadata

Description

This lesson opens the chain's first link: the catalog. You already installed it in module 1's lesson 4 — a SqlCatalog backed by SQLite, kiosko_catalog.db — but you never literally looked at what it stores inside. This lesson opens that database with a direct SQL query, without going through the PyIceberg API, and shows the catalog is much simpler than its name suggests: it doesn't contain a single row of Kiosko data, nor the table's schema, nor the list of snapshots — it contains, only, one row with one piece of text: the exact path to the metadata file that is, right now, "the current one."

Connection to the module. Lesson 1 of this module presented the catalog as "the court clerk's desk" in the court-case-file analogy — the one that tells you which cover sheet to go to, without you having to look it up by hand. This lesson makes that claim literal: you're going to read, with your own eyes, the exact row in kiosko_catalog.db that plays that role.

An analogy: the court clerk's desk doesn't store case files, it stores addresses

Continuing the module's analogy. The court clerk's desk, if you ask it about the case "Kiosko vs. Nobody" (a made-up name, just for the analogy), doesn't hand you a copy of the whole file from its own desk — it tells you: "the current file is in the archive, hallway 3, shelf B, folder with today's stamp." The desk stores an address, not the content. If a new version of the file gets filed tomorrow, the desk updates that address — "it's now in the folder with tomorrow's stamp" — but today's folder still exists in hallway 3, with nobody having destroyed it. This lesson shows, in kiosko_catalog.db, exactly that same structure: an address, not content.

Worked example: the exact row that registers kiosko.fact_orders

Step 1 — Query kiosko_catalog.db directly, without PyIceberg

kiosko_catalog.db is a normal SQLite database — any tool that speaks SQL can open it, with no need for PyIceberg at all. This lesson uses Python's standard library sqlite3 module, precisely to show that no special magic is needed:

# inspect_catalog_db.py
import os
import sqlite3

catalog_db_path = os.path.abspath("kiosko_catalog.db")

conn = sqlite3.connect(catalog_db_path)
cur = conn.cursor()

cur.execute("SELECT name FROM sqlite_master WHERE type='table';")
print("Internal tables in kiosko_catalog.db:", [r[0] for r in cur.fetchall()])

cur.execute(
    "SELECT catalog_name, table_namespace, table_name, metadata_location, "
    "previous_metadata_location FROM iceberg_tables;"
)
row = cur.fetchone()
catalog_name, namespace, table_name, metadata_location, previous_metadata_location = row

print("\n=== kiosko.fact_orders row in iceberg_tables ===")
print("catalog_name:", catalog_name)
print("table_namespace:", namespace)
print("table_name:", table_name)
print("metadata_location:")
print(" ", metadata_location)
print("previous_metadata_location:")
print(" ", previous_metadata_location)

conn.close()

What to expect (verified by running the actual script, against the kiosko_catalog.db left by module 1; the absolute paths are going to match your own working directory, not this text's):

Internal tables in kiosko_catalog.db: ['iceberg_tables', 'iceberg_namespace_properties']

=== kiosko.fact_orders row in iceberg_tables ===
catalog_name: kiosko
table_namespace: kiosko
table_name: fact_orders
metadata_location:
  file:///.../kiosko_warehouse/kiosko/fact_orders/metadata/00001-<uuid>.metadata.json
previous_metadata_location:
  file:///.../kiosko_warehouse/kiosko/fact_orders/metadata/00000-<uuid>.metadata.json

Notice the two columns that really matter: metadata_location points to the metadata file with the highest number — 00001-..., the one that registered the first snapshot in module 1's lesson 6; previous_metadata_location points to the earlier file — 00000-..., the one that module's lesson 5 created when the table was still empty, with current_snapshot() is None. Both files still exist on disk, neither one deleted — lesson 6 of this module confirms it with a real ls. The catalog, literally, only keeps track of "which one is current" and "which one was current just before that" — nothing more.

Step 2 — Confirm the same thing, now with the PyIceberg API

PyIceberg exposes the same information without you having to write SQL by hand:

# via_api.py
import os

from pyiceberg.catalog import load_catalog

warehouse_path = os.path.abspath("kiosko_warehouse")
catalog_db_path = os.path.abspath("kiosko_catalog.db")

catalog = load_catalog(
    "kiosko", type="sql",
    uri=f"sqlite:///{catalog_db_path}", warehouse=f"file://{warehouse_path}",
)
table = catalog.load_table("kiosko.fact_orders")

print("table.metadata_location:")
print(" ", table.metadata_location)

What to expect (verified by running the actual script):

table.metadata_location:
  file:///.../kiosko_warehouse/kiosko/fact_orders/metadata/00001-<uuid>.metadata.json

The exact same value you read directly from SQLite in step 1 — table.metadata_location isn't different information, it's the same row from iceberg_tables, exposed through a more convenient API. catalog.load_table("kiosko.fact_orders") did, under the hood, exactly what you did by hand in step 1: query iceberg_tables, read metadata_location, and use that path to open the metadata file — lesson 3's job.

Diagram: what the catalog contains, and what it does NOT contain

flowchart LR
    subgraph DB["kiosko_catalog.db (SQLite)"]
        ROW["row: kiosko.fact_orders\nmetadata_location -> 00001-...json\nprevious_metadata_location -> 00000-...json"]
    end
    ROW -->|"points to"| CURRENT["00001-<uuid>.metadata.json\n(current)"]
    ROW -.->|"used to point to"| OLD["00000-<uuid>.metadata.json\n(previous, still on disk)"]

There's no column at all in iceberg_tables for schema, snapshots, or partition-spec — all three of those live inside the JSON file metadata_location points to. This separation is deliberate: changing which metadata file is "the current one" is a cheap operation — updating a single piece of text in one SQLite row — while the metadata file itself can grow with every new snapshot without the catalog ever having to rewrite anything about its own structure.

Going deeper: why this is what makes concurrency control possible

It's worth pausing on why the catalog stores only a path, not the table's full content. If two processes try to write to kiosko.fact_orders at the same time, each one prepares its own new metadata file without touching anyone else's — that's cheap and requires no coordination; the moment of truth happens when each process tries to update the metadata_location row in iceberg_tables to point to its own new file. A catalog backed by a real SQL database — like this guide's SqlCatalog — can use a transaction with a condition ("update metadata_location to my new file, only if it still says what I read when I started") to guarantee that, if both processes compete, exactly one wins and the other gets a clear error to retry. This is, precisely, the guarantee module 1's lesson 4 called optimistic concurrency control — and now you can see, in iceberg_tables's minimal structure, why it's such a simple operation to protect: it's a single UPDATE over a single piece of text, not a rewrite of any data table.

Common mistakes

Looking for the table's schema or data directly inside kiosko_catalog.db. What happens: someone, after seeing that iceberg_tables exists, expects to find columns there with names like order_id, store_id, etc., or even rows with Kiosko data. Why it happens: "catalog" sounds, to someone coming from a traditional database, like the place everything lives. How to spot it: if your SQL query against kiosko_catalog.db looks for a schema or data column and doesn't find it, revisit this lesson's worked example — iceberg_tables has exactly five columns, none related to the table's content. How to fix it: the schema lives in the metadata file (lesson 3); the data lives in Parquet files (lesson 5); the catalog only stores the address to the first one.

Editing previous_metadata_location by hand, thinking it "cleans up" the catalog. What happens: someone, seeing previous_metadata_location points to a file from an old version, edits or manually deletes it in the SQLite database, assuming it's accumulated junk. Why it happens: the name "previous" sounds like something no longer needed. How to spot it: if after touching kiosko_catalog.db by hand, catalog.load_table("kiosko.fact_orders") starts failing or behaving unexpectedly, check whether you modified that column directly. How to fix it: never edit kiosko_catalog.db by hand outside a read-only query like this lesson's — previous_metadata_location is part of how PyIceberg detects and prevents race conditions; the real cleanup of old metadata is an explicit operation (expire_snapshots, module 7), not a manual edit of a catalog row.

Assuming every table in the kiosko namespace shares a single row in iceberg_tables. What happens: someone, after creating more tables in later modules of this guide (dim_product, dim_store, etc.), expects to see all the information together in one row, instead of one row per table. Why it happens: it's easy to think of "Kiosko's catalog" as a single unit. How to spot it: if you run SELECT COUNT(*) FROM iceberg_tables after creating several tables and expected to see 1, check the primary key (catalog_name, table_namespace, table_name) in iceberg_tables's CREATE TABLE. How to fix it: every Iceberg table — kiosko.fact_orders, and every table you add in future modules — has its own independent row, with its own metadata_location; the kiosko namespace is only a logical grouping (module 1's lesson 5), not a merger of its tables in the catalog.

Exercises

Exercise 1 — Reproduce both queries yourself. On your own machine, with module 1's kiosko_catalog.db and kiosko_warehouse/ available, run this lesson's sqlite3 script and its PyIceberg API script. Confirm that metadata_location (from both) and table.metadata_location are exactly the same text.

See solution

If your kiosko.fact_orders table is exactly as module 1 left it (a single table.append()), both scripts should return the same path, ending in 00001-<uuid>.metadata.json. If your path ends in 00000-..., it means your table never received module 1's lesson 6 load — check that you ran that step before starting this module.

Exercise 2 — Explain, in your own words, the difference between metadata_location and previous_metadata_location. Without looking at the Going deeper section yet, write 2-3 sentences explaining what each column stores and why both exist, not just one.

See solution

metadata_location is the address to the metadata file that's current right now — the one any new reader should use. previous_metadata_location is the address to the file that was current just before the latest update — a single snapshot of history, stored directly in the catalog, useful mainly for detecting and debugging race conditions between concurrent writes (this lesson's Going deeper). The complete history of every earlier snapshot doesn't live here — that would be too many rows for a single column — it lives, instead, inside the current metadata file itself, in its snapshots[] list (lesson 3).

Exercise 3 — Prediction: what would happen if you deleted kiosko_catalog.db but left kiosko_warehouse/ intact? Without testing it yet, predict: if you deleted the kiosko_catalog.db file (the catalog), but kept the kiosko_warehouse/ folder with all its metadata, manifest, and data files, could you still query kiosko.fact_orders with catalog.load_table("kiosko.fact_orders")?

See solution

No — load_catalog("kiosko", type="sql", uri=f"sqlite:///{catalog_db_path}", ...) would recreate a completely new, empty kiosko_catalog.db (SQLite creates the file if it doesn't exist), with no rows at all in iceberg_tables, so catalog.load_table("kiosko.fact_orders") would fail with a table-not-found error, even though the data and metadata files still exist intact in kiosko_warehouse/. This confirms, with an extreme case, this lesson's Going deeper section: the catalog isn't a copy of the table's information — it's the only place that knows where to find it. Without it, the data is still there, but nobody knows, without guessing, which metadata file is the current one.

Summary and next step

In this lesson you opened kiosko_catalog.db directly with SQL, without going through PyIceberg, and confirmed the catalog stores exactly five columns per table — none with Kiosko's schema or data — with metadata_location as the central piece: the address to the current metadata file. You confirmed, with table.metadata_location, that the PyIceberg API exposes the same information, with no difference at all.

Before moving on you should be able to: explain what an iceberg_tables row contains and what it does NOT contain; and explain why separating "the address" from "the content" is what makes concurrency control possible.

Lesson 3 follows the arrow: it finally opens the file metadata_location points to — the metadata file itself, readable JSON, with the schema, the partitioning, and the list of snapshots this minimalist catalog never stored directly.

Resources

• PyIceberg — API reference, Table.metadata_location and Catalog.load_table(). py.iceberg.apache.org/api. In English.
• Apache Iceberg — official documentation, catalog specification (Catalog), the guarantee that a metadata-pointer update is atomic. iceberg.apache.org/spec. In English.
• This guide's DESIGN doc — the choice of SqlCatalog/SQLite and the optimistic concurrency control guarantee, already introduced in module 1's lesson 4. src/guides/lakehouse-and-iceberg-guide/DISENO.md. In Spanish.