Hidden partitioning: the same query, without knowing the layout

Description

Lesson 2 left you with a real filing cabinet, on disk, and an uncomfortable lesson: to read it correctly, every reader has to be told what the folder convention is. This lesson asks the opposite question: what happens if you ask the same business question — "give me S01's data" — to an Iceberg table? You're going to use kiosko.fact_orders, the table that has already existed since this guide's module 1, to answer it — without creating any new table yet, without touching partitioning yet (that table, as module 2 confirmed, has a completely empty PartitionSpec). That's precisely what makes the result compelling: the syntax you're going to use here is exactly the same you're going to use in lesson 5 against a really partitioned table.

Connection to the module. This lesson is the module's hinge: lesson 2 showed the visible folder's cost; this lesson shows the absence of that cost on Iceberg's side. Lessons 4 through 7 build the real partitioning that makes that absence of cost also, additionally, fast — but the way you ask isn't going to change a single line from here on.

An analogy: the mail carrier, again, before he even has bags

Going back to lesson 1's mail carrier: this lesson is the moment before the carrier organizes anything. He still doesn't have bags by resident or by day — he just has a pile of mail, with no particular physical order. And even so, when you ask him for "Ana's mail," he knows how to find it — he checks what he has, filters by the name you gave him, and hands you the correct result. It's not as fast as it would be with organized bags — he has to check more than strictly necessary — but the way you ask him is already, from this moment, the final one: you're never going to have to tell him "check bag 3." That's exactly what you're about to confirm with kiosko.fact_orders.

Worked example: the same question, with no folder at all

Step 1 — Load the table, and confirm it has no partitioning

# l3_hidden_partitioning_demo.py
import os

from pyiceberg.catalog import load_catalog

warehouse_path = os.path.abspath("kiosko_warehouse")
catalog_db_path = os.path.abspath("kiosko_catalog.db")
catalog = load_catalog(
    "kiosko", type="sql",
    uri=f"sqlite:///{catalog_db_path}", warehouse=f"file://{warehouse_path}",
)
table = catalog.load_table("kiosko.fact_orders")

print("kiosko.fact_orders -- current PartitionSpec:", table.spec())

What to expect (verified by running the real script, with kiosko.fact_orders from module 1 already loaded):

kiosko.fact_orders -- current PartitionSpec: []

Confirmed, again: [], exactly what module 2 (lesson 3) already showed. There's no PartitionField at all — no physical drawer distinct from "all the files together."

Step 2 — The hidden query: filter by store_id, with no file mentioned

# the SAME syntax you're going to use in lesson 5 against fact_orders_at_scale,
# really partitioned -- this query's code never mentions a folder
s01 = table.scan(row_filter="store_id == 'S01'").to_arrow()
s01_revenue = sum(s01.column("revenue").to_pylist())
print(f"row_filter=\"store_id == 'S01'\" -> {s01.num_rows} rows, revenue={round(s01_revenue, 2)}")

What to expect:

row_filter="store_id == 'S01'" -> 16 rows, revenue=38.3

16 rows, 38.3 in revenue — the same S01 breakdown you already know from data-engineering-foundations-guide. And notice what's not in this call: no path, no folder name, no partitioning="hive" argument. row_filter="store_id == 'S01'" is an expression over a business column's value, period — the same kind of predicate you'd use in a SQL WHERE, with no reference at all to how the files are organized internally.

Step 3 — Count how many files the scan actually touched, even though there's nothing to prune yet

plan = list(table.scan(row_filter="store_id == 'S01'").plan_files())
print(f"data files the scan actually touched: {len(plan)}")

What to expect:

data files the scan actually touched: 1

A single file — because kiosko.fact_orders, with its forty rows, never had more than one. Without partitioning, there's nothing to prune yet: any query, filtered or not, touches the only file that exists. This isn't a flaw in the example — it's precisely why this lesson uses a table without partitioning: it isolates the question "does the query syntax change?" (answer: no) from the question "does performance change?" (answer: yes, and lesson 5 is going to measure it with a table that does have files to prune).

Diagram: the same call, two different tables

flowchart TB
    Q["table.scan(row_filter=\"store_id == 'S01'\").to_arrow()"]
    Q -->|"against kiosko.fact_orders\n(no partitioning, this lesson)"| A["1 file touched out of 1 total\nnothing to prune yet"]
    Q -->|"against kiosko.fact_orders_at_scale\n(partitioned, lesson 5)"| B["1 file touched out of 3 total\nreal, measurable pruning"]
    A -.->|"SAME line of code\nin both cases"| B

Going deeper: who decides the layout, now

Lesson 2 left an open question: if knowledge of the layout no longer lives in every reader, where does it live? The answer, with this table as minimal evidence, is: in the table itself, not in the reader or the writer. When someone wrote table.append() on kiosko.fact_orders in module 1, they didn't have to decide "I'm going to organize this by store_id" — that decision, if it existed, would be in the table's PartitionSpec, automatically consulted on every append() and every scan(). Since the spec is empty, there's no decision to make — but the mechanism is identical to what you're going to see in lesson 5, with a real spec: neither whoever writes nor whoever reads needs to repeat, in their own code, what the partitioning criterion is. That's the precise, complete definition of hidden partitioning: not that partitioning doesn't exist, but that its existence (or its absence, as in this case) is transparent to any code that queries or writes the table, beyond the business filter you'd already write anyway.

Common mistakes

Thinking row_filter needs to know, in advance, whether the table is partitioned by that column. What happens: someone, on writing row_filter="store_id == 'S01'" against a table without partitioning, expects it to fail or behave differently than against a partitioned table. Why it happens: if you're coming from systems where filtering by a partition column requires special syntax (like lesson 2's explicit partitioning="hive"), it's natural to expect something similar here. How to spot it: if your code changes depending on whether the table you're querying is partitioned or not, revisit this lesson — lesson 5's step 2 and step 3 use the identical line of code. How to fix it: row_filter always accepts any expression over schema columns, whether the table is partitioned or not — the only observable difference is how many files plan_files() touches underneath, never the syntax above.

Confusing plan_files() with to_arrow(), and expecting both to return the same kind of result. What happens: someone tries to iterate over table.scan(...).plan_files() expecting to see data rows, and is surprised to see scan task objects (FileScanTask) instead of S01's rows. Why it happens: both methods hang off the same scan(), so it's easy to assume they do the same thing under a different name. How to spot it: if your code expects business columns (store_id, revenue) when iterating plan_files(), revisit this lesson's step 3 — there, only len(plan) gets counted, no rows from that result are ever read. How to fix it: to_arrow() (or to_pandas(), to_pylist()) materializes the rows that satisfy the filter; plan_files() returns the list of files the engine decided it needs to open to answer that query — it's the piece that lets you measure, with a number, how much pruning the scan did.

Exercises

Exercise 1 — Reproduce the hidden query yourself, against kiosko.fact_orders. With module 1's table available, run this lesson's three steps. Confirm current PartitionSpec: [], 16 rows and 38.3 in revenue for S01, and 1 file touched.

Exercise 2 — Filter by product_id instead of store_id, and confirm the syntax doesn't change. Write table.scan(row_filter="product_id == 'P002'").to_arrow() against the same table, and confirm how many rows and what revenue you get.

p002 = table.scan(row_filter="product_id == 'P002'").to_arrow()
p002_revenue = sum(p002.column("revenue").to_pylist())
print(f"P002: {p002.num_rows} rows, revenue={round(p002_revenue, 2)}")

Exercise 3 — Prediction: what would happen if you tried to filter by a column that doesn't exist? Without running it, predict: if you wrote table.scan(row_filter="region == 'LATAM'") — a column kiosko.fact_orders never had — do you expect it to silently return zero rows, or to fail with an error?

Summary and next step

In this lesson you asked the same business question as lesson 2 — "give me S01's data" — against an Iceberg table, without partitioning yet, and confirmed the syntax never mentions a file or a folder: table.scan(row_filter="store_id == 'S01'"). You verified, with plan_files(), that today there's nothing to prune — a single file, always touched — and established that this same line of code is the one you're going to reuse, with no change at all, against a really partitioned table.

Before moving on you should be able to: write a row_filter over any column of the schema; explain the difference between to_arrow() and plan_files(); and anticipate that lesson 5 is going to repeat this same query, but this time with real evidence of file pruning.

Before repeating that query against a partitioned table, you need precise vocabulary for how an Iceberg partition's layout gets decided. Lesson 4 gives you the three transforms you're going to use for the rest of this module.

Resources

• PyIceberg — API reference, table.scan(row_filter=...)'s and table.scan(...).plan_files()'s exact syntax. py.iceberg.apache.org/api. In English.
• Apache Iceberg — official documentation, "Partitioning," "Iceberg's hidden partitioning" section (the formal definition this lesson demonstrates with code). iceberg.apache.org/docs/latest/partitioning. In English.
• This guide's DESIGN doc — module 5's section, with the explicit rule that the query should never mention the physical structure. src/guides/lakehouse-and-iceberg-guide/DISENO.md. In Spanish.