Inspecting Kiosko's table on disk

Description

Lessons 2 through 5 walked the whole chain with Python: direct SQL over the catalog, json.load() over the metadata, PyIceberg's table.inspect.manifests()/table.inspect.files() over the manifests and the data. This lesson repeats that same walkthrough once more, but this time from the terminal, with ls, file, and cat — without a single line of Python. The goal is for you to see, with your own eyes and with no layer of abstraction in between, the physical difference between a readable file and one that isn't, and to confirm, by counting files by hand, that nothing module 1 wrote has disappeared.

Connection to the module. This lesson doesn't teach any new concept — it's the same chain from lessons 2 through 5, verified with generic operating-system tools, so an Iceberg table's anatomy stops feeling like "something only PyIceberg can see" and starts feeling like what it is: a real file structure, in a real directory, that any shell tool can walk.

An analogy: walking the archive hallway, in person

So far, every lesson had you ask the librarian (the catalog or the PyIceberg API) for a specific piece of the case file. This lesson is different: you walk the archive hallway yourself, open the shelf with your own hands, and look at the folders one by one. You don't need anyone to hand you anything prepared — you just count how many folders there are, read the ones you can read at a glance, and confirm that the ones you can't read really can't be read without specialized help.

Worked example: the whole warehouse, from the terminal

Step 1 — List the complete tree, with sizes

ls -la kiosko_warehouse/kiosko/fact_orders/data/
ls -la kiosko_warehouse/kiosko/fact_orders/metadata/

What to expect (the UUID-based names are your own run's; the number of files and their type are deterministic):

kiosko_warehouse/kiosko/fact_orders/data/:
00000-0-<uuid>.parquet   3.2K

kiosko_warehouse/kiosko/fact_orders/metadata/:
00000-<uuid>.metadata.json                     1.1K
00001-<uuid>.metadata.json                     2.1K
<uuid>-m0.avro                                  4.7K
snap-<snapshot_id>-0-<uuid>.avro                1.8K

Count with this listing: one data file, two JSON metadata files, one manifest file, one manifest list — five files total, for a table with a single snapshot. Notice the two .metadata.json files: 00000-... (1.1K, the empty table from module 1's lesson 5) and 00001-... (2.1K, larger because it already includes the full snapshots[] section) — both still exist, neither was deleted after that module's lesson 6 load. This is the whole guide's most direct, simplest piece of evidence that "nothing gets overwritten": a plain ls, with no Iceberg code at all, already confirms it.

Step 2 — file: identify each file's real type, without trusting the extension alone

file kiosko_warehouse/kiosko/fact_orders/metadata/*.json \
     kiosko_warehouse/kiosko/fact_orders/metadata/*.avro \
     kiosko_warehouse/kiosko/fact_orders/data/*.parquet

What to expect:

kiosko_warehouse/kiosko/fact_orders/metadata/00000-<uuid>.metadata.json: JSON data
kiosko_warehouse/kiosko/fact_orders/metadata/00001-<uuid>.metadata.json: JSON data
kiosko_warehouse/kiosko/fact_orders/metadata/<uuid>-m0.avro: Apache Avro version 1
kiosko_warehouse/kiosko/fact_orders/metadata/snap-<snapshot_id>-0-<uuid>.avro: Apache Avro version 1
kiosko_warehouse/kiosko/fact_orders/data/00000-0-<uuid>.parquet: Apache Parquet

The Unix file command doesn't trust the file name's extension — it reads each file's first bytes and recognizes its real format from its internal signature. This output confirms, independently of any naming convention, exactly what lessons 3, 4, and 5 of this module already demonstrated with code: the two .metadata.json files really are JSON, the two .avro files really are Avro, and the .parquet really is Parquet. No file lies about what it is.

Step 3 — Try reading the metadata as text (works), then the manifest (doesn't work)

head -c 220 kiosko_warehouse/kiosko/fact_orders/metadata/00001-<your-uuid>.metadata.json

What to expect:

{"location":"file:///.../kiosko_warehouse/kiosko/fact_orders","table-uuid":"<uuid>",...

Immediately readable — it's exactly the same text json.load() parsed with no trouble in lesson 3. Now, the same attempt on the manifest file:

head -c 220 kiosko_warehouse/kiosko/fact_orders/metadata/<your-uuid>-m0.avro

What to expect (the characters ^A, ^N, ^L represent non-printable control bytes — this text is an approximate transcription of what you'd see in your own terminal):

Obj^A^N^Lschema<non-printable-byte>^G{"type":"struct","fields":[{"id":1,"name":"order_id","type":"string","required":true},{"id":2,"name":"store_id","type":"string","required":true},{"id":3,"name":"product_id","type":"strin

Here's this lesson's central piece of evidence, live: the first bytes (Obj) and a fragment of the header — the embedded Avro schema, in JSON, describing the manifest's structure — really are readable, because Avro stores its own schema as text at the start of the file. But as soon as that header ends, the rest of the file — the manifest's actual data, compressed with the deflate codec — stops being recognizable text. This isn't a corrupted file: it's exactly the expected behavior of a compressed binary format, the same kind of format you already accepted without question in a Parquet file, now visible with your own eyes in an Avro manifest.

Diagram: readable vs. binary, confirmed from the terminal

flowchart TB
    subgraph LEGIBLE["Readable with head/cat (plain text)"]
        A["00000-...metadata.json (1.1K)"]
        B["00001-...metadata.json (2.1K)"]
    end
    subgraph BINARIO["Binary (Avro/Parquet, need a specific reader)"]
        C["snap-...avro (1.8K) -- manifest list"]
        D["...-m0.avro (4.7K) -- manifest file"]
        E["00000-0-...parquet (3.2K) -- data"]
    end
    LEGIBLE -.->|"json.load() (lesson 3)"| OK1["works directly"]
    BINARIO -.->|"table.inspect.* / pq.read_table() (lessons 4-5)"| OK2["works with the right reader"]
    BINARIO -.->|"cat/head as text"| FAIL["readable header,\nunreadable body"]

Going deeper: why each file's size tells a story

It's worth reading step 1's sizes carefully, because they aren't arbitrary. 00000-...metadata.json (1.1K) is smaller than 00001-...metadata.json (2.1K) because the first one describes an empty table — with no entry at all in snapshots[] — while the second already carries the full first-snapshot section, with its nine-field summary. The manifest file (4.7K) is larger than the manifest list (1.8K) because the manifest file stores, for every data file it enumerates, detailed per-column statistics — the lower_bounds/upper_bounds/null_value_counts you saw as table.inspect.files() columns in lesson 5 — while the manifest list only needs a summary per manifest file, much more compact. And the data file (3.2K) is, of the five, the one holding the greatest amount of real information — forty complete rows, seven columns — but thanks to Parquet's columnar compression, it ends up comparable in size to the manifest file that only describes one data file. None of these sizes is a coincidence: every layer in the chain is designed to be proportional to the amount of information it needs to summarize, not to the amount of actual data underneath.

Common mistakes

Running these commands from the wrong directory, and seeing "No such file or directory." What happens: someone runs ls kiosko_warehouse/... from a directory different from the one used for module 1, and the command fails because that relative path doesn't exist there. Why it happens: unlike this module's Python code, which uses os.path.abspath() to resolve paths regardless of the working directory (the pattern taught in module 1's lesson 4), this lesson's shell commands use literal relative paths, which do depend on where you're standing. How to spot it: if ls kiosko_warehouse/... fails but your lesson 2 Python script (which uses the same path, resolved to absolute) works fine, check which folder you're running the shell command from. How to fix it: cd into the same working directory where you ran module 1's scripts before running this lesson's commands — or use the full absolute path in every command.

Interpreting an Avro file's readable header as evidence that "it's actually text after all." What happens: someone, seeing the readable JSON fragment at the start of the manifest file in step 3, concludes the whole file is, deep down, plain text with some noise. Why it happens: seeing recognizable JSON creates a false sense of familiarity. How to spot it: if you try to parse the whole Avro file with json.load() (as you did with the metadata in lesson 3), you're going to get an immediate syntax error, right after the header ends. How to fix it: the schema header is, on purpose, the only part of an Avro file meant to be visually inspected by a human — it helps debug schema problems with no tools; the body with the actual data is compressed and structured in binary, and requires a real Avro reader (like the one table.inspect.manifests() uses internally) to decode.

Counting files with ls and not noticing the manifest file and manifest list count differs from table.inspect.manifests(). What happens: someone, running ls over the metadata/ folder, manually counts the .avro files and expects that number to match table.inspect.manifests().num_rows directly. Why it happens: in Kiosko's case with a single snapshot, both numbers do match (there's one manifest list and one manifest file, and table.inspect.manifests() returns 1 row) — so the confusion goes unnoticed in this module, but it can lead to a wrong intuition later on. How to spot it: remember ls counts files on disk, including manifest lists; table.inspect.manifests() specifically counts manifest files, a subset of what you see in ls. How to fix it: to tell them apart at a glance in the ls listing, use the naming pattern: a file starting with snap- is a manifest list; a file ending in -m<number>.avro is a manifest file — exercise 2 of this lesson practices exactly that distinction.

Exercises

Exercise 1 — Reproduce the full walkthrough yourself, with no Python. In your own terminal, with module 1's kiosko_warehouse/ available, run this lesson's three steps in order: ls -la, file, and the two head -c 220. Confirm you count five files total, and that the .metadata.json head is completely readable while the .avro one turns unreadable after the header.

See solution

If your table has the same state module 1 left, you should count exactly five files: one .parquet, two .metadata.json, one -m0.avro manifest file, and one snap-*.avro manifest list. If file isn't available on your system (uncommon, but possible in some minimal environments), you can confirm the same result by checking each file's first bytes with head -c 10 <file> | xxd — JSON files start with {, Avro files start with Obj, and Parquet files start with PAR1.

Exercise 2 — Tell the manifest list apart from the manifest file using only the naming pattern. Without running table.inspect.manifests() again, look at this lesson's step 1 ls output and, using only the naming pattern explained in the third common mistake, identify which of the two .avro files is the manifest list and which is the manifest file.

See solution

The file starting with snap-<snapshot_id>-0-<uuid>.avro is the manifest list — the snap- prefix followed by the snapshot_id is the pattern lesson 3 already identified inside the manifest-list field of the metadata file. The file ending in <uuid>-m0.avro, without the snap- prefix, is the manifest file — the -m0 suffix indicates it's the first (and only, in this case) manifest file from that write.

Exercise 3 — Prediction: what shell command would you use to confirm, with no Python, that the data file is valid Parquet and not just "something that ends in .parquet"? Without using pq.read_table() (you already did that in lesson 5), predict which command from this lesson independently confirms the data file is real Parquet.

See solution

file kiosko_warehouse/kiosko/fact_orders/data/*.parquet — the file command doesn't trust the .parquet extension in the name; it reads the file's real binary signature (the PAR1 magic bytes at the start and end of the file, part of the Parquet spec) and independently confirms it's "Apache Parquet," with no need for a full format reader. It's the same kind of check — binary signature, not extension — step 2 of this lesson already used for the .avro and .metadata.json files.

Summary and next step

In this lesson you walked the whole kiosko_warehouse/ from the terminal, without a single line of Python: you counted exactly five files (one data, two metadata, one manifest list, one manifest file), confirmed each one's real type with file, and saw, with your own eyes, that an Avro manifest has a readable header followed by an unreadable binary body without the right reader — the physical evidence, with no Iceberg code involved at all, of everything lessons 2 through 5 already demonstrated with Python.

Before moving on you should be able to: list and count, from the terminal, every file in a freshly loaded Iceberg table; use file to confirm a file's real type without trusting its extension; and explain why an Avro file's header is partially readable even though its body isn't.

You walked the whole chain twice: once with Python, link by link (lessons 2 through 5), and once from the terminal, at a glance (this lesson). Lesson 7 goes back to Python, but now focused on a different angle: not "what's inside each file," but "how the snapshot changes over time" — using PyIceberg's four inspection methods (table.history(), table.inspect.snapshots(), table.inspect.manifests(), table.inspect.files()) together, as the toolkit you're going to use constantly for the rest of this guide.

Resources

• Unix file command — manual documentation (man file), the file-type-identification-by-binary-signature tool used in this lesson. Available on any Unix/Linux/macOS system. In English (system manual pages).
• Apache Avro — Object Container Files format specification, the section explaining the embedded-schema header this lesson observed partially readable. avro.apache.org/docs/++version++/specification/#object-container-files. In English.
• This guide's DESIGN doc — the explicit instruction to explore the on-disk warehouse/ as a verifiable part of module 2. src/guides/lakehouse-and-iceberg-guide/DISENO.md. In Spanish.