File format vs. table format

Description

This lesson takes the conceptual step that makes everything that follows understandable: the precise, unambiguous distinction between a file format (what Parquet is) and a table format (what Iceberg is). These aren't two alternatives competing to solve the same problem — they're two different layers, one on top of the other, and confusing them is the most common source of misunderstandings about what Iceberg is and isn't.

Connection to the module. Lesson 2 showed, with evidence, four times the Kiosko ecosystem needed something a file format, on its own, doesn't give. This lesson names precisely what that something is. Lessons 4 through 8 of this module install and use Iceberg for the first time; this lesson is the last conceptual stop before writing code.

An analogy: the box of loose photos and the album with an index, again, in more detail

Lesson 1 of this module introduced the analogy: a Parquet file is like a box of loose photos, well developed, but with no index organizing them; an Iceberg table is like an album with an index up front. It's worth taking that analogy one step further, because the exact distinction matters.

Notice something important: the photos themselves — the paper, the developing, the sharpness — are exactly the same in both cases. The album didn't develop the photos again with a different technique; it simply added an index to them. In the same way, an Iceberg table's data files are Parquet, period — the same columnar, compressed, typed format you already used in the six previous guides. Iceberg doesn't replace how a row's bytes are stored; it adds, on top, a layer that knows: how many photos there are right now, which ones there were yesterday, in what order they were added, and how to reconstruct any earlier version without anyone having had to save it separately on purpose.

That's, precisely, this lesson's definition: a file format describes how the bytes of a set of rows are stored. A table format describes how those files are organized, versioned, and queried as if they were a single logical entity called a "table."

Worked example: the same question, answered by a file and by a table

Lesson 1's worked example showed that a loose Parquet file, read with pyarrow.parquet.read_table(), can tell you how many rows it has and with what schema — right now, nothing more. Compare that answer with the one PyIceberg gives about the table you're going to build in lessons 5 and 6 of this same module (preview: the full code and install come in lesson 4; here only the shape of the question and the answer is shown, to contrast them):

# the question a Parquet file answers
import pyarrow.parquet as pq
pa_table = pq.read_table("fact_orders.parquet")
print(pa_table.num_rows)          # 40 -- but "40, according to which version?"

# the same question, answered by an Iceberg table
table = catalog.load_table("kiosko.fact_orders")
print(table.scan().to_arrow().num_rows)      # 40 -- the CURRENT version
print(table.history())                        # the FULL list of versions
print(table.scan(snapshot_id=snap_v1).to_arrow().num_rows)  # 40 -- a SPECIFIC version, by id

The first question — pa_table.num_rows — only has one possible answer, because there's only one version: the one in the file, right now, with no way to ask for another. The second question — table.scan().to_arrow().num_rows — has the same numeric answer today (40), but it's an answer of a different kind: it's the answer to "how many rows does the current version have?", a question that admits a third form — table.scan(snapshot_id=snap_v1) — that a loose Parquet file can't formulate or answer, because it has no notion of "a version other than the current one." You don't need to fully understand the syntax of table.history() or snapshot_id yet — that's, precisely, module 3's content — what this comparison shows, at this exact point in the guide, is the difference in available vocabulary: a file can only talk about "what's there"; a table can talk about "what's there, what was there, and how we got from one to the other."

The six capabilities a table format adds, and where you'll see them

Iceberg adds, on top of the same Parquet as always, six concrete capabilities that no loose file can give you on its own. You're not going to build all six in this module — each has its own dedicated module later in this guide — but it's worth naming all of them here, once, precisely:

1. Real ACID transactions. A write never leaves the table halfway — either it applied completely, or it didn't apply at all. It contrasts directly with the risk window in lesson 2's overwrite-partition. (Covered in depth in module 4.)
2. Automatic snapshots on every commit. Every write creates a new, immutable version, with no one having to declare a history column. (Module 3.)
3. Time travel. Querying any earlier snapshot by its identifier, reconstructing the table's exact past with no valid_from/valid_to written by anyone. (Module 3.)
4. Safe schema evolution. Adding, renaming, or dropping a column without rewriting a single existing data file. (Module 4.)
5. Hidden partitioning and partition evolution. Queries filter by a business column, never by folder structure; the partition scheme can be changed going forward without touching data already written. (Module 5.)
6. Native MERGE INTO / upserts. The same "update without losing history" problem data-modeling solved by hand and dbt automated with columns, now solved as an operation native to the format. (Module 6.)

Notice that the six capabilities aren't scattered ideas — each one answers, by name, one of the four problems from lesson 2. Capabilities 2 and 3 answer problems 2 and 3 (dim_product history). Capability 1 answers problem 1 (overwrite-partition). Capability 5 answers problem 4 (Hive folders). And capability 6 puts the P002 change problem back on the table, this time solved natively instead of with columns.

Diagram: two layers, one on top of the other

flowchart TB
    subgraph TABLA["TABLE format (Iceberg) -- the something on top"]
        T1["Catalog: points to current metadata"]
        T2["Metadata: schema, snapshots, partitioning"]
        T3["ACID / snapshots / time travel /\nschema evolution / hidden partitioning / MERGE"]
    end
    subgraph ARCHIVO["FILE format (Parquet) -- unchanged"]
        A1["Columnar encoding"]
        A2["Compression (snappy/zstd)"]
        A3["Data types per column"]
    end
    TABLA -->|"organizes, versions,\nand points to"| ARCHIVO

Going deeper: why this distinction isn't just semantic

It's tempting to treat "file format vs. table format" as a vocabulary difference with no practical consequences. It isn't, and it's worth saying why with a concrete example: if tomorrow someone asks you "compare this week's revenue against last week's, exactly as each one looked the moment it ended," with a loose Parquet the only way to answer is to have saved, on purpose, a separate copy of each week — a week_2026_08_03/ folder, another week_2026_08_10/ — a manual discipline someone has to maintain forever, with no room for error. With an Iceberg table, that question is answered with two snapshot_id numbers, captured automatically by the engine at the exact moment of each write — nobody had to remember anything, because the table format already did it by design. That difference — permanent manual discipline, versus the format's automatic guarantee — is, precisely, what separates "a file format used well" from "a real table format." Module 3 of this guide builds exactly that example, with the real P002 change.

Common mistakes

Thinking "table" is just a directory with several Parquet files inside. What happens: someone, seeing that an Iceberg table lives, on disk, as a folder with data/ and metadata/ subfolders, concludes Iceberg is simply "a folder-organizing convention," similar to the Hive partitioning from problem 4 of lesson 2. Why it happens: visually, both things are folders with files inside. How to spot it: if you think you could reconstruct Iceberg's behavior simply by organizing your own Parquet files into conventionally named subfolders, you're missing the central piece — it's not the folder organization that makes the difference, it's the metadata file (JSON, which module 2 of this guide inspects in depth) that records, with transactional guarantees, which version is current and which ones came before. How to fix it: all of module 2 is dedicated to this exact distinction — you're going to open, yourself, the warehouse/ directory this module is going to create, and you're going to see that what makes the difference isn't where the files live, but what points to what.

Believing you have to choose between "using Parquet" or "using Iceberg." What happens: someone understands this lesson's comparison as a choice between two competing technologies, the same way you'd choose between two databases. Why it happens: the language of "format A vs. format B" invites, by habit, thinking in terms of mutually exclusive alternatives. How to spot it: if you're asking yourself "should I use Parquet or Iceberg for this project?", the question is framed wrong — revisit this lesson's diagram: Iceberg uses Parquet as its data file format, it doesn't compete with it. How to fix it: the right question is "does this Parquet need to behave like a table (with history, with safe evolution, with transactional guarantees), or is a well-written file enough?" — in lesson 6 of this module you're going to see, in code, the same fact_orders.parquet as always turn into the Iceberg table without a single byte of how its rows are encoded ever changing.

Exercises

Exercise 1 — Classify each capability. For each of the following, say whether it's a file format (Parquet) capability or a table format (Iceberg) capability: (a) columnar compression of a numeric column; (b) querying what the whole table looked like three weeks ago; (c) per-column data types within a single file; (d) adding a new column without rewriting the existing data.

See solution

(a) file format — compression (snappy, zstd) is a property of how Parquet encodes a column's bytes, independent of whether that Parquet lives loose or inside an Iceberg table. (b) table format — it's exactly time travel, a capability that depends on a snapshot history existing, something a loose file doesn't have. (c) file format — the per-column typed schema is a native property of Parquet, present even in a completely loose .parquet file, with no catalog at all. (d) table format — safe schema evolution depends on the table format knowing how to interpret old files (without the new column) and new files (with it) as a single coherent table.

Exercise 2 — Explain in your own words why Iceberg "doesn't develop the photos again." Returning to the album analogy, explain in 2-3 sentences why it's correct to say Iceberg doesn't change how the data is encoded inside each Parquet file.

See solution

Iceberg adds a layer of organization and versioning on top of the Parquet files — the catalog and the metadata chain — but each individual data file is still a normal .parquet, written with the same columnar encoding, the same compression, and the same types you already used in the previous guides. The concrete proof is in lesson 6 of this module: the same fact_orders.parquet reconstructed with pyarrow gets loaded into the Iceberg table with table.append(), with no intermediate step "developing the photos again" with a different technique — an index just gets added to them.

Exercise 3 — Prediction: what happens if you open an Iceberg data file directly with pandas? Without having installed Iceberg yet, predict: if in lesson 6 of this module you opened, with pandas.read_parquet(), one of the data files Iceberg wrote inside warehouse/kiosko/fact_orders/data/ directly, skipping the catalog entirely, would you expect it to work? Why?

See solution

Yes, it should work with no error at all — because, as this lesson explains, a data file belonging to an Iceberg table is a completely normal .parquet, readable with any tool that already knows how to read Parquet, with no need for the catalog or any piece of Iceberg. What you won't get by reading it this way is the table context: you won't know whether that specific file belongs to the current snapshot or an earlier one, and you won't be able to ask it for a different version — for that you do need to go through the catalog and the PyIceberg API, exactly as this lesson's worked example shows.

Summary and next step

In this lesson you precisely defined the central distinction of the whole guide: a file format (Parquet) describes how the bytes of a set of rows are encoded; a table format (Iceberg) describes how those files are organized, versioned, and queried as a single logical entity. You saw the six concrete capabilities Iceberg adds, and the map of which module of this guide builds each one.

Before moving on you should be able to: explain, without using the word "better," how a file format differs from a table format; and name Iceberg's six capabilities, even without the technical detail of each one yet.

With this conceptual foundation in place, lesson 4 leaves theory behind for the rest of the module: it installs PyIceberg for real, on your own machine, and creates Kiosko's first local catalog.

Resources

• Apache Iceberg — official documentation, "What is Iceberg?", the formal table-format definition and its relationship to the file formats it organizes. iceberg.apache.org/docs/latest. In English.
• Apache Parquet — official documentation, the specification of the columnar file format Iceberg uses unmodified. parquet.apache.org/docs. In English.
• PyIceberg — API reference, the exact syntax of table.scan(), table.history(), and table.scan(snapshot_id=...) used in this lesson's worked example — covered in depth starting in module 3. py.iceberg.apache.org/api. In English.
• This guide's DESIGN doc — the full map of the six capabilities and which module builds each one. src/guides/lakehouse-and-iceberg-guide/DISENO.md. In Spanish.