The data files: the Parquet you already know

Description

This lesson opens the chain's last link: the data file the manifest file from lesson 4 enumerates. And, unlike the three previous links, this one needs no special Iceberg tool to open — it's Parquet, exactly the same columnar format you already used in data-engineering-foundations-guide, in spark-and-distributed-processing-guide, and in this guide's own module 1, lesson 6. This lesson closes the loop module 1's lesson 1 opened: "Iceberg sits on top of Parquet, it never replaces it" — and here's the direct proof, opening the real data file with pyarrow, with no PyIceberg involved at all.

Connection to the module. Lessons 2 through 4 walked through three layers of indirection, each more specialized than the last. This lesson finally reaches the end of the chain — and the end turns out to be the most familiar part of the whole module.

An analogy: the actual photos, with no more layers of indirection

After following the cover sheet, the evidence index, and opening the right evidence folder, you finally hold the photo itself in your hand. There's no further layer — it's the evidence, as is. You can look at it, zoom in on it, compare it with another one, without having to consult any further index. That is, precisely, what an Iceberg data file is: the end of the chain, with no additional indirection — a normal Parquet file, which any tool that speaks Parquet can open, exactly the way it would open any other .parquet you've seen in this ecosystem.

Worked example: inspecting the data file, then opening it directly

Step 1 — table.inspect.files(): which data file the manifest file enumerates

# inspect_data_files.py
import os

import pyarrow.parquet as pq
from pyiceberg.catalog import load_catalog

warehouse_path = os.path.abspath("kiosko_warehouse")
catalog_db_path = os.path.abspath("kiosko_catalog.db")

catalog = load_catalog(
    "kiosko", type="sql",
    uri=f"sqlite:///{catalog_db_path}", warehouse=f"file://{warehouse_path}",
)
table = catalog.load_table("kiosko.fact_orders")

print("=== table.inspect.files() -- one row per data file ===")
files = table.inspect.files().select(
    ["file_path", "file_format", "record_count", "file_size_in_bytes"]
)
row = files.to_pylist()[0]
data_file_path = row["file_path"]
print("  file_path:", os.path.basename(data_file_path))
print("  file_format:", row["file_format"])
print("  record_count:", row["record_count"])
print("  file_size_in_bytes:", row["file_size_in_bytes"])
print("\nnumber of data files:", files.num_rows)

What to expect (the file name includes a UUID assigned in your run; record_count and the rest of the business values are deterministic):

=== table.inspect.files() -- one row per data file ===
  file_path: 00000-0-<uuid>.parquet
  file_format: PARQUET
  record_count: 40
  file_size_in_bytes: 3273

number of data files: 1

table.inspect.files() explicitly confirms, in the file_format column, something you'd so far only inferred from the file extension: PARQUET. record_count: 40 matches, number for number, table.scan().to_arrow().num_rows from module 1's lesson 7 — lesson 4's manifest file pointed to exactly this file, and this file contains exactly Kiosko's forty rows for the week.

Step 2 — Open that same file directly with pq.read_table(), without PyIceberg

print("\n=== Opening THAT SAME file directly with pq.read_table() ===")
local_path = data_file_path.replace("file://", "")
direct = pq.read_table(local_path)
print("num_rows:", direct.num_rows)
print("schema:")
print(direct.schema)
print("\nfirst 3 rows, selected columns:")
sample = direct.select(["order_id", "store_id", "product_id", "revenue"]).slice(0, 3).to_pylist()
for r in sample:
    print(" ", r)

What to expect:

=== Opening THAT SAME file directly with pq.read_table() ===
num_rows: 40
schema:
order_id: string not null
  -- field metadata --
  PARQUET:field_id: '1'
store_id: string not null
  -- field metadata --
  PARQUET:field_id: '2'
product_id: string not null
  -- field metadata --
  PARQUET:field_id: '3'
quantity: int32 not null
  -- field metadata --
  PARQUET:field_id: '4'
unit_price: double not null
  -- field metadata --
  PARQUET:field_id: '5'
revenue: double not null
  -- field metadata --
  PARQUET:field_id: '6'
order_ts: timestamp[us] not null
  -- field metadata --
  PARQUET:field_id: '7'

first 3 rows, selected columns:
  {'order_id': 'ORD-1001', 'store_id': 'S01', 'product_id': 'P001', 'revenue': 1.65}
  {'order_id': 'ORD-1002', 'store_id': 'S01', 'product_id': 'P002', 'revenue': 1.2}
  {'order_id': 'ORD-1003', 'store_id': 'S02', 'product_id': 'P003', 'revenue': 1.5}

Stop on this result, because it's the whole lesson's central point: pq.read_table() — the same pyarrow.parquet function you already used in module 1's lesson 1, with no PyIceberg dependency at all — opens this file with no problem, and returns the actual forty rows, with ORD-1001 and ORD-1002 recognizable, exactly as you wrote them. This is the complete data file, with no additional layer of indirection. And notice one more thing, in the printed schema: every column carries, in its Parquet metadata, a PARQUET:field_id — 1 for order_id, 2 for store_id, and so on. These are exactly the same field_ids you declared with NestedField in module 1's lesson 5, and the same ones you read in schemas[0]["fields"] from the metadata file in this module's lesson 3 — Iceberg doesn't invent a new mapping at every level of the chain, it reuses the same field_id all the way from the declared schema to the physical Parquet file.

Diagram: the end of the chain, with no further layer

flowchart LR
    A["catalog\n(lesson 2)"] --> B["metadata.json\n(lesson 3)"]
    B --> C["manifest list\n(lesson 4)"]
    C --> D["manifest file\n(lesson 4)"]
    D --> E["data file\n00000-0-<uuid>.parquet\n(this lesson)"]
    E -.->|"direct pq.read_table(),\nno PyIceberg"| F["pyarrow.Table\n40 rows, same schema\nas always"]

Going deeper: why this matters for any tool outside Iceberg

It's worth noting the practical consequence of the final file being standard Parquet, with no proprietary extension at all: any engine that knows how to read Parquet can read an Iceberg table's raw data, without understanding absolutely anything about catalogs, manifest lists, or snapshots. A pandas script, a quick exploration notebook, or even a completely different guide from this ecosystem could open 00000-0-<uuid>.parquet directly and get correct data — with one important warning: that direct read skips every guarantee Iceberg exists to provide. If the table had several data files, some from an old write already replaced by an overwrite() (module 3), reading a single file by hand could show you stale or incomplete data, with no warning at all — exactly the same risk module 1's lesson 1 already named about a loose Parquet. The correct way to read an Iceberg table always goes through the catalog and the current snapshot (table.scan().to_arrow()), which guarantees reading exactly the set of files the current snapshot declares as current — never more, never less. This lesson opened the file directly only for educational inspection purposes, not as a recommended practice for reading production data.

Common mistakes

Assuming opening the data file directly is a valid way to read the table in production. What happens: someone, after seeing in this lesson that pq.read_table() works fine, starts reading individual Parquet files directly instead of going through table.scan(). Why it happens: it works, technically, in the simple case of a single file and a single snapshot — the risk only becomes visible once the table has more than one file or more than one snapshot. How to spot it: if your code reads files with glob("*.parquet") over an Iceberg table's data/ folder, instead of using table.scan(), revisit this lesson's Going deeper section. How to fix it: always use table.scan().to_arrow() (or the filtered variants you already used in module 1's lesson 7) to read data from an Iceberg table — that's the only way that respects the guarantee of "reading exactly the current snapshot, no more, no fewer files."

Being surprised that file_size_in_bytes (from table.inspect.files()) and the real size on disk don't match to the byte exactly. What happens: someone compares file_size_in_bytes against the size ls -la shows for the same file, finds a tiny difference, and worries about corruption. Why it happens: in most cases both numbers match exactly — as in this lesson's example — but they can differ if the filesystem rounds the reported size to blocks, or if there's some capture difference between the moment of the write and the moment of the query. How to spot it: if the difference is only a few bytes or matches your filesystem's block size, it isn't corruption. How to fix it: nothing to fix in the typical case — if you want the exact, reliable size of a file, file_size_in_bytes (recorded in the manifest file itself at write time) is the correct source, more reliable than trusting the filesystem after the fact.

Not connecting the PARQUET:field_id seen in this lesson with the field_id from module 1's lesson 5. What happens: someone sees PARQUET:field_id: '1' in the schema printed by pq.read_table() and treats it as an internal Parquet detail with no connection to anything else. Why it happens: it shows up in a different print format (PyArrow's field metadata) than the NestedField(field_id=1, ...) you already saw. How to spot it: if you can't explain why order_id has field_id: 1 both in the Parquet file and in the JSON metadata file, revisit module 1's lesson 5 Going deeper section and this lesson's step 2 side by side. How to fix it: it's exactly the same number, propagated by Iceberg all the way from the Schema you declared to the physical Parquet file itself — it's the piece that makes it possible for a RENAME COLUMN (module 4) to not have to rewrite any data file: the field_id embedded in the Parquet never changes, only the label the catalog presents it with changes.

Exercises

Exercise 1 — Reproduce both steps yourself. On your own machine, with module 1's state available, run this lesson's full script. Confirm record_count (from table.inspect.files()) and num_rows (from directly reading with pq.read_table()) are both 40.

See solution

If your table has the single data file module 1 left, both numbers should be 40, with no difference at all — they're, literally, two different ways of counting the same physical file's rows: one through the metadata Iceberg registered at write time, the other by reading the file directly and actually counting. Their matching confirms Iceberg's metadata is faithful to the file's real content.

Exercise 2 — Compare table.scan().to_arrow() against a direct pq.read_table(), column by column. Using module 1's state, run table.scan().to_arrow().schema (the correct way to read, via Iceberg) and compare it against direct.schema from step 2 of this lesson (the direct file read). Do you find any difference?

See solution

There shouldn't be any visible difference at this point in the guide — same seven columns, same types, same field_ids — because only one data file and one snapshot exist; in this case, reading "via Iceberg" and reading "the file directly" produce exactly the same result. The real difference would only show up once the table had more than one data file with different schema versions (after a schema evolution, module 4) or with files from an already-replaced write (after an overwrite(), module 3) — there, table.scan() would correctly filter only the current snapshot's files, while reading loose files by hand could mix data from different versions with no warning at all.

Exercise 3 — Prediction: what would happen if you opened a manifest file (.avro) with pq.read_table(), as if it were Parquet? Without testing it yet, predict: if you passed the path of a manifest file (*-m0.avro) to pq.read_table() instead of a real data file's path, what do you expect to happen?

See solution

It should fail with an error — something related to an invalid file format or an incorrect "magic number" — because pq.read_table() specifically expects Parquet's binary format, which starts and ends with the magic bytes PAR1; an Avro file has its own, completely different binary structure (it starts with the bytes Obj\x01, visible in lesson 6 of this module). This is, in spirit, the same mistake you'd make trying to open a .png image with a .jpg reader — both are binary, but with incompatible internal structures. The correct lesson for reading a manifest file is table.inspect.manifests() (lesson 4), never pq.read_table().

Summary and next step

In this lesson you opened the chain's final link: the data file the manifest file from lesson 4 enumerates turned out to be exactly the same Parquet you already know — file_format: PARQUET, record_count: 40 — and you opened it directly with pq.read_table(), with no PyIceberg dependency at all, confirming the field_id embedded in the Parquet matches, number for number, the one declared in module 1's lesson 5.

Before moving on you should be able to: explain why the final data file needs no special Iceberg tool to open; and explain the risk of reading data files directly instead of going through table.scan().

You walked the whole chain, one link at a time: catalog → metadata → manifest list → manifest files → data files. Lesson 6 repeats this same walkthrough, but now complete, from the terminal, with real shell commands — so you see, with your own eyes, the difference between what's readable and what's binary, without any Python script's help.

Resources

• Apache Parquet — official file format documentation, the same reference you already used in the six previous guides in the ecosystem. parquet.apache.org/docs/. In English.
• PyIceberg — API reference, table.inspect.files() and the exact columns it returns, including per-column metrics (readable_metrics). py.iceberg.apache.org/api. In English.
• Apache Iceberg — official documentation, "Table Spec," "Data Files" section, the formal definition of what a DataFile must register inside a manifest. iceberg.apache.org/spec. In English.
• This guide's DESIGN doc — M1L1's central claim ("Iceberg sits on top of Parquet, it never replaces it"), confirmed here with real code. src/guides/lakehouse-and-iceberg-guide/DISENO.md. In Spanish.