Verifying the same 106.15 total

Description

Loading forty rows isn't, by itself, proof that the migration to Iceberg was correct — there could be forty rows with corrupted data, with mixed-up columns, or with a wrongly calculated revenue, and the count would still say 40. This lesson does the verification that actually matters: reading kiosko.fact_orders from Iceberg and confirming, with the same revenue per store and the same total (106.15) that six different engines already confirmed across the six previous guides, that the migration preserved the exact data, not just the row count.

Connection to the module. This lesson closes the arc of lessons 4, 5, and 6 — install, create, load — with the question that actually matters: is the data still correct? It's the same verification discipline you already saw in every previous guide in the ecosystem, now applied for the first time to an Iceberg table.

An analogy: counting the photos, and then looking at whether they're the right photos

A careless archivist, receiving a new album, might settle for counting the pages: "forty photos, matches what I was told, done." A rigorous archivist does something more: besides counting, they look at a sample of the photos and confirm they're really the right photos — not just any forty photos, but exactly the forty orders for the week of August 3 to 9, with the correct revenue on each one. This lesson is that second step: you don't settle for len(table.scan().to_arrow()) == 40 — you already confirmed that in lesson 6 — you're going to add up the revenue, broken down by store, and compare that number against the same one six different engines already verified before Iceberg.

Worked example: the same verification as always, now over Iceberg

# verify_fact_orders_iceberg.py
import os

from pyiceberg.catalog import load_catalog

DIM_STORE = [
    {"store_id": "S01", "store_name": "Kiosko Centro", "city": "Bogota"},
    {"store_id": "S02", "store_name": "Kiosko Norte", "city": "Lima"},
    {"store_id": "S03", "store_name": "Kiosko Sur", "city": "Santiago"},
]
store_names = {s["store_id"]: s["store_name"] for s in DIM_STORE}

warehouse_path = os.path.abspath("kiosko_warehouse")
catalog_db_path = os.path.abspath("kiosko_catalog.db")

catalog = load_catalog(
    "kiosko", type="sql",
    uri=f"sqlite:///{catalog_db_path}", warehouse=f"file://{warehouse_path}",
)
table = catalog.load_table("kiosko.fact_orders")

scanned = table.scan().to_arrow()
total_rows = scanned.num_rows
total_revenue = round(sum(scanned.column("revenue").to_pylist()), 2)

print("=== kiosko.fact_orders (Iceberg) ===\n")
print(f"len(table.scan().to_arrow()) = {total_rows}")

by_store = scanned.group_by("store_id").aggregate([("revenue", "sum")])
print("\n=== Revenue by store ===")
for row in sorted(by_store.to_pylist(), key=lambda r: r["store_id"]):
    sid = row["store_id"]
    print(f"{sid} {store_names[sid]:<14}: revenue={round(row['revenue_sum'], 2)}")

print(f"\nTotal week revenue: {total_revenue}")

What to expect (verified by running the actual script, against the table loaded in lesson 6):

=== kiosko.fact_orders (Iceberg) ===

len(table.scan().to_arrow()) = 40

=== Revenue by store ===
S01 Kiosko Centro : revenue=38.3
S02 Kiosko Norte  : revenue=38.8
S03 Kiosko Sur    : revenue=29.05

Total week revenue: 106.15

Stop on these five numbers, because they're exactly the same ones data-engineering-foundations-guide, python-for-data-engineering-guide, data-modeling-for-analytics-guide, dbt-analytics-engineering-guide, airflow-and-declarative-orchestration-guide, and spark-and-distributed-processing-guide already verified — six guides, six different engines (plain Python with CSV, a package installed with uv, DuckDB, DuckDB via dbt, Airflow orchestrating that same package, and a PySpark DataFrame) — and now a seventh confirmation, reading from a real Iceberg table for the first time. S01=38.3, S02=38.8, S03=29.05, total 106.15 — the same data, with zero deviation, after passing through seven completely different implementations.

Additional verification: the grain holds up over Iceberg too

data-modeling-for-analytics-guide (module 1) declared fact_orders's grain as "one order line," verified with COUNT(*) against COUNT(DISTINCT order_id || '-' || product_id). It's worth repeating that same verification here, now with pyarrow.compute over the data read from Iceberg, to confirm the migration didn't introduce a single duplicate row:

import pyarrow.compute as pc

keys = pc.binary_join_element_wise(scanned.column("order_id"), scanned.column("product_id"), "-")
distinct_keys = pc.count_distinct(keys)

print("total_rows =", scanned.num_rows)
print("distinct_order_product_lines =", distinct_keys.as_py())

qty_invalid = scanned.filter(pc.field("quantity") <= 0).num_rows
print("rows with invalid quantity =", qty_invalid)

What to expect (verified by running the actual script):

total_rows = 40
distinct_order_product_lines = 40
rows with invalid quantity = 0

40 against 40 — the grain declared in data-modeling-for-analytics-guide holds up exactly the same over the Iceberg table, with no duplicate row introduced by the migration. And zero rows with an invalid quantity, the same quality check validate_orders() already guaranteed in data-engineering-foundations-guide, well before this data ever reached Iceberg.

Diagram: seven engines, one same number

flowchart LR
    A["foundations\nPython + CSV"] --> G["106.15"]
    B["python-for-data-engineering\nkiosko_pipeline (uv)"] --> G
    C["data-modeling\nDuckDB"] --> G
    D["dbt\nDuckDB via dbt"] --> G
    E["airflow\nkiosko_pipeline orchestrated"] --> G
    F["spark\nPySpark DataFrame"] --> G
    H["this guide\nPyIceberg + Iceberg"] --> G

Going deeper: why this verification is stronger than "trusting the migration"

It's tempting to think that, if table.append() didn't throw any error in lesson 6, the migration must necessarily have been correct — after all, Iceberg validates the schema on every write, as you saw in that lesson's common mistakes. But validating the schema (correct types and column names) isn't the same as validating the value of each data point — a bug in build_fact_orders_parquet.py that calculated revenue = quantity + unit_price instead of quantity * unit_price would have produced forty rows with a perfectly valid schema, with no Iceberg error at all, and with a total completely different from 106.15. This lesson is the one that catches that kind of error — it doesn't trust "if there was no exception, it's fine"; it recalculates the business number from scratch, and compares it against an external, already known value, independently verified six times before. This is, precisely, the same discipline every previous guide in this ecosystem demanded before accepting any new table as good — Iceberg doesn't change that discipline, it only changes where the data being verified lives.

Common mistakes

Trusting only the row count, without adding up the revenue. What happens: someone, seeing len(table.scan().to_arrow()) == 40 in lesson 6, considers the migration finished, without running this lesson's revenue verification. Why it happens: the row count is the simplest possible check, and it feels like enough evidence. How to spot it: if your verification process stops at "forty rows, done," revisit this lesson's Going deeper section — forty rows of corrupted data are still forty rows. How to fix it: always compare a business number (revenue, in this case) against an already known external value — this lesson's table against the six previous guides is exactly that external value for Kiosko's case.

Adding revenue directly in Python with sum() over a long list, and finding tiny rounding differences. What happens: someone manually sums the revenue values with standard floating-point arithmetic, and gets something like 106.14999999999999 instead of exactly 106.15, and worries there's a data error. Why it happens: binary floating-point arithmetic (IEEE 754), the same used by Python, DuckDB, and Spark, doesn't represent decimal numbers like 0.55 or 1.20 exactly — it's a known limitation of any language using double/float64, not a bug in this guide or in Iceberg. How to spot it: if your total has more than two visible decimals with several 9s in a row near the end, it's floating-point rounding, not a data error. How to fix it: this lesson's worked example uses round(total_revenue, 2) exactly for this reason — rounding to two decimals at the end of the calculation is the standard practice for presenting money amounts, and it's the same technique the six previous guides in this ecosystem already used, without exception.

Exercises

Exercise 1 — Reproduce both verifications yourself. With kiosko.fact_orders already loaded (lesson 6), run this lesson's revenue-by-store script and its grain script. Confirm you get a total of 106.15 and 40 == 40 in the grain check.

Exercise 2 — Calculate revenue by product, not just by store. Using scanned (the result of table.scan().to_arrow() from this lesson's worked example), write the code that groups by product_id instead of store_id, and confirm which of Kiosko's four products generated the most revenue for the week.

by_product = scanned.group_by("product_id").aggregate([("revenue", "sum")])
for row in sorted(by_product.to_pylist(), key=lambda r: r["product_id"]):
    print(f"{row['product_id']}: revenue={round(row['revenue_sum'], 2)}")

Exercise 3 — Explain why this verification doesn't depend on Iceberg at all. In 2-3 sentences, explain why this lesson's code — adding up revenue, grouping by store_id, comparing against an external value — would be exactly the same, line for line after table.scan().to_arrow(), if the data came from a pandas.DataFrame read directly from a CSV.

Summary and next step

In this lesson you verified, with the same revenue per store (S01=38.3, S02=38.8, S03=29.05) and the same total (106.15) that six different engines already confirmed across the six previous guides, that migrating fact_orders to Iceberg preserved the exact data — not just the row count. And you repeated data-modeling-for-analytics-guide's grain check (40 == 40, no duplicates) over the data read from the Iceberg table.

Before moving on you should be able to: explain why counting rows isn't enough evidence of a correct migration; reproduce both of this lesson's verifications on your own machine; and recite from memory Kiosko's revenue per store (38.3/38.8/29.05, total 106.15) — a number you're going to keep seeing for the rest of this guide.

With Kiosko's first Iceberg table loaded and verified, lesson 8 — this module's closing project — brings lessons 4 through 7 together into a single end-to-end flow, and formally closes the promise this module opened in lesson 1.

Resources

• PyIceberg — API reference, table.scan() and its direct interoperability with pyarrow.Table. py.iceberg.apache.org/api. In English.
• Apache Arrow — official pyarrow.compute documentation, the aggregation functions (group_by, aggregate) used in this lesson. arrow.apache.org/docs/python/compute.html. In English.
• data-modeling-for-analytics-guide DESIGN doc — source of the grain declaration (COUNT(*) vs. COUNT(DISTINCT order_id || '-' || product_id)) this lesson repeats over Iceberg. src/guides/data-modeling-for-analytics-guide/DISENO.md. In Spanish.
• This guide's DESIGN doc — Kiosko's canonical number (106.15, S01=38.3/S02=38.8/S03=29.05) verified, for the seventh time, in this lesson. src/guides/lakehouse-and-iceberg-guide/DISENO.md. In Spanish.