Verifying the same forty rows arrive

Description

Lesson 6 read Kiosko's seven files and reported orders_df.count() == 40. This lesson doesn't just trust that number on its own — it verifies it from a second angle, with a completely different tool (pure Python, csv.DictReader, the same one foundations used), and then digs deeper into the verification: how many rows came from each of the seven files, how many orders each store has, and whether any null value slipped through after applying the typed schema. It's the same "count twice, with two different methods" discipline you already saw in foundations (assert counts == counts_again) and in data-modeling (before == after after every JOIN), now applied to Spark.

Connection to the module. This lesson closes out the reading work from lessons 5 and 6 with cross-evidence. Lesson 8 reuses every check from this lesson inside the module's closing mini-project.

An analogy: closing out the register, counted twice

A cashier closing out their shift doesn't trust a single number. They count the physical cash in the register, and separately check the summary the sales system generated during the shift — two completely independent methods that, in theory, should arrive at the same total. If they match, they have real evidence the shift closed correctly. If they don't, something needs an explanation before closing out. This lesson does exactly that with Kiosko: it counts orders with Spark (orders_df.count()) and, separately, with pure Python and csv.DictReader — the same method foundations used from the start. If both numbers agree on 40, you have real evidence, not just one tool's word, that Spark read exactly what it should have.

Worked example: cross-verification, by file, by store, and by nulls

# verify_kiosko_orders.py
import csv
import glob

from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from pyspark.sql.types import (
    StructType, StructField, StringType, IntegerType, DoubleType, TimestampType,
)

spark = SparkSession.builder.appName("kiosko-spark").master("local[*]").getOrCreate()

orders_schema = StructType([
    StructField("order_id", StringType(), False),
    StructField("store_id", StringType(), False),
    StructField("product_id", StringType(), False),
    StructField("quantity", IntegerType(), False),
    StructField("unit_price", DoubleType(), False),
    StructField("order_ts", TimestampType(), False),
])
orders_df = spark.read.csv(
    "orders_2026-08-*.csv", schema=orders_schema, header=True, enforceSchema=False,
)

print("=== Verification 1: cross-count, Spark vs pure Python ===")
spark_count = orders_df.count()

python_count = 0
for path in sorted(glob.glob("orders_2026-08-*.csv")):
    with open(path, newline="") as f:
        python_count += sum(1 for _ in csv.DictReader(f))

print(f"spark_count = {spark_count}")
print(f"python_count (csv.DictReader) = {python_count}")
assert spark_count == python_count == 40
print("Verification: spark_count == python_count == 40 -> OK\n")

print("=== Verification 2: rows by source file ===")
by_file = (
    orders_df
    .withColumn("source_file", F.element_at(F.split(F.input_file_name(), "/"), -1))
    .groupBy("source_file")
    .count()
    .orderBy("source_file")
)
by_file.show(truncate=False)

print("=== Verification 3: orders by store ===")
orders_df.groupBy("store_id").count().orderBy("store_id").show()

print("=== Verification 4: nulls by column (should all be 0) ===")
null_counts = orders_df.select([
    F.sum(F.col(c).isNull().cast("int")).alias(c) for c in orders_df.columns
])
null_counts.show()

spark.stop()

What to expect. Running python3 verify_kiosko_orders.py, the output is exactly this:

=== Verification 1: cross-count, Spark vs pure Python ===
spark_count = 40
python_count (csv.DictReader) = 40
Verification: spark_count == python_count == 40 -> OK

=== Verification 2: rows by source file ===
+---------------------+-----+
|source_file          |count|
+---------------------+-----+
|orders_2026-08-03.csv|8    |
|orders_2026-08-04.csv|6    |
|orders_2026-08-05.csv|2    |
|orders_2026-08-06.csv|5    |
|orders_2026-08-07.csv|7    |
|orders_2026-08-08.csv|9    |
|orders_2026-08-09.csv|3    |
+---------------------+-----+

=== Verification 3: orders by store ===
+--------+-----+
|store_id|count|
+--------+-----+
|     S01|   16|
|     S02|   13|
|     S03|   11|
+--------+-----+

=== Verification 4: nulls by column (should all be 0) ===
+--------+--------+----------+--------+----------+--------+
|order_id|store_id|product_id|quantity|unit_price|order_ts|
+--------+--------+----------+--------+----------+--------+
|       0|       0|         0|       0|         0|       0|
+--------+--------+----------+--------+----------+--------+

Four independent checks, all four correct. Verification 2 confirms, file by file, the exact counts you already know from the comment in raw_orders.py in data-modeling (8, 6, 2, 5, 7, 9, 3 — Monday through Sunday). Verification 3 confirms, store by store, the same numbers you already saw in the foundations capstone and in module 1 of data-modeling: S01=16, S02=13, S03=11. Verification 4 confirms lesson 6's explicit schema didn't leave any value incorrectly converted — not a single column has even one NULL.

Diagram: four angles of verification converging on the same data

flowchart TD
    A["orders_df (Spark, lesson 6)\n40 rows"] --> V1["Verification 1:\nSpark.count() vs\ncsv.DictReader\n40 == 40"]
    A --> V2["Verification 2:\nrows by file\n8+6+2+5+7+9+3 = 40"]
    A --> V3["Verification 3:\nrows by store\n16+13+11 = 40"]
    A --> V4["Verification 4:\nnulls by column\nall at 0"]
    V1 --> R["Confidence in orders_df:\nnot just 'count()==40',\nfour independent pieces of evidence"]
    V2 --> R
    V3 --> R
    V4 --> R

Going deeper: why four checks, not just count()

It's tempting to think that, if orders_df.count() == 40 already matches what's expected, nothing else is needed. But count() on its own, exactly as data-modeling-for-analytics-guide warned about COUNT(*), can't catch every possible problem. A correct total count is compatible with several silently broken scenarios: you could have forty rows where there should really be forty-one (one real row lost, and another duplicated by mistake, exactly canceling out); you could have the correct forty rows but with a mistyped column producing NULL in every row without the total count reflecting it at all; you could have the right files but read with the column order swapped, as you saw in lesson 6's common mistake — the total count would still be 40 even though the data was completely misaligned.

That's why this lesson doesn't stop at verification 1. Verification 2 (by file) would catch it if some file got read too many or too few times — for example, if a badly written wildcard pattern captured a file it shouldn't have, or skipped one. Verification 3 (by store) cross-checks the result against a number you already know by heart from three previous guides — if S01 showed, say, 15 instead of 16, you'd immediately know something changed relative to the original data, even if the total still came out to 40 because of an offsetting error in another store. And verification 4 (nulls) is the only way to confirm lesson 6's typed schema actually converted every value with no gaps left behind. Four independent checks, each able to catch a kind of problem the other three wouldn't see.

Common mistakes

Stopping at count() == 40 and checking nothing else. What happens: someone runs orders_df.count(), sees 40, and considers the data read closed with no further checks. Why it happens: the number matches what's expected, and that feels like enough confirmation. How to spot it: if your only evidence the read was correct is a single total number, you can't rule out the offsetting scenarios described in the "going deeper" section — lost and duplicated rows canceling each other out, for example. How to fix it: apply at least one additional check that breaks the total down into known parts — by file, by store, by any dimension you already have the correct number for from a previous guide — the way this lesson's verifications 2 and 3 did.

Checking nulls on the wrong columns, or not checking them at all. What happens: someone trusts that, since the schema declared False on every StructField (not nullable), Spark automatically rejects any null value, and so never runs an explicit check. Why it happens: StructField("order_id", StringType(), False) looks, at first glance, like a hard guarantee the column will never have NULL. How to spot it: StructField's third argument (nullable) is, in practice, more a statement of intent than an enforced validation when reading a CSV — an empty value in the source file can end up as NULL in the DataFrame even if you declared nullable=False, with no error thrown by Spark on read. How to fix it: don't rely on the schema's nullable parameter as if it were an active check — verify nulls explicitly, as this lesson's verification 4 did, especially before any downstream calculation (like the revenue = quantity * unit_price you're going to build in module 3, where a NULL in quantity or unit_price would silently produce a NULL in the result).

Comparing against numbers "from memory" without confirming the memory is correct. What happens: someone compares the result of groupBy("store_id").count() against a number they vaguely remember from an earlier guide, without checking the exact source, and concludes "it matches" or "it doesn't match" with less certainty than they think they have. Why it happens: trusting memory is faster than going back to look up the exact number in the earlier guide. How to spot it: if your cross-check depends on a number you can't point to precisely in an earlier document or lesson, it isn't a real verification — it's a second hunch, not independent evidence. How to fix it: this lesson's numbers (S01=16, S02=13, S03=11) are anchored, precisely, in module 1 of data-modeling-for-analytics-guide (lesson 5, exercise 2) and in the data-engineering-foundations-guide capstone — if you ever doubt a reference number, go back to that exact source before declaring a verification successful.

Exercises

Exercise 1 — Verify the count by product, not just by store. Extend the worked example with a fifth check: orders_df.groupBy("product_id").count(), sorted by product_id. The expected numbers, per foundations, are P001: 16, P002: 10, P003: 7, P004: 7.

print("=== Verification 5: orders by product ===")
orders_df.groupBy("product_id").count().orderBy("product_id").show()

=== Verification 5: orders by product ===
+----------+-----+
|product_id|count|
+----------+-----+
|      P001|   16|
|      P002|   10|
|      P003|    7|
|      P004|    7|
+----------+-----+

Exercise 2 — Confirm quantity is never zero or negative. Without using groupBy, write a check confirming no row in orders_df has quantity <= 0 — the same class of quality check validate_orders() already guaranteed in foundations, now confirmed again on Spark's DataFrame.

invalid_quantity_count = orders_df.filter(F.col("quantity") <= 0).count()
print(f"Rows with invalid quantity: {invalid_quantity_count}")
assert invalid_quantity_count == 0

Rows with invalid quantity: 0

Exercise 3 — Explain, without code, why the cross-check (Spark vs pure Python) is stronger than repeating the same query in Spark twice. In 2-3 sentences, explain why comparing orders_df.count() against csv.DictReader is a more reliable check than simply running orders_df.count() twice in a row in Spark.

Summary and next step

This lesson didn't read any new data — it took lesson 6's orders_df and subjected it to four independent checks: cross-count against pure Python, rows by source file, orders by store, and nulls by column. All four matched exactly what was expected, confirming — with evidence, not just one tool's word — that Spark read exactly the same week of forty orders you already know from foundations, python-for-data-engineering, and data-modeling.

Before moving on you should be able to: explain why count() == 40 alone isn't sufficient evidence; recite the counts by store from memory (S01=16, S02=13, S03=11); and explain why a cross-check with a different tool (pure Python versus Spark) is more reliable than repeating the same query with the same tool.

Lesson 8 — the closing mini-project — pulls the installation, the session, the read, and these four checks together into a single delivery script: Kiosko's first Spark session, verified end to end.

Resources

• Apache Spark — column functions (pyspark.sql.functions), including input_file_name(), used in this lesson's verification 2 to trace which file each row came from. spark.apache.org/docs/latest/api/python/reference/pyspark.sql/functions.html.
• data-modeling-for-analytics-guide DESIGN doc — the source of the counts by store (S01=16, S02=13, S03=11) this lesson verifies again with Spark. src/guides/data-modeling-for-analytics-guide/DISENO.md
• data-engineering-foundations-guide DESIGN doc — the source of the counts by product and of validate_orders(), the original quality gate that guarantees quantity > 0. src/guides/data-engineering-foundations-guide/DISENO.md