Reading `orders`, `dim_store`, and `dim_product` with an explicit schema

Description

Module 1 read a single table — orders — with an explicit StructType. This lesson does the same thing three times: orders again, and for the first time in this guide, dim_store and dim_product, each with its own schema, different from the other two. By the end of this lesson you're going to have the three raw pieces lesson 3 is going to join into a single fact_orders — but nothing joined yet, exactly like in module 1 you read orders before calculating anything with it.

Connection to the module. This lesson is the module's starting point: without the three tables read correctly, with the correct types, no .join() in lesson 3 is going to work as expected. A store_id read as IntegerType in one table and as StringType in another, for example, would make lesson 3's JOIN find no matches at all — silently, with no error. This lesson exists so that doesn't happen.

An analogy: three shelves in the same archive, three different card formats

In module 1 you got to know Kiosko's public archive: seven folders, all with the same card format — order_id, store_id, product_id, quantity, unit_price, order_ts — that you could read as a single collection. That same archive actually has three different shelves, not one. The first — the one you already know — holds order cards, seven folders, one format. The second shelf holds three cards, one per store: name, city. The third holds four cards, one per product: name, category, unit cost. Each shelf has its own card format, different from the other two, and confusing one shelf's format with another's — reading a store card as if it were a product card — produces, at best, an immediate error, and at worst, misaligned data with no warning at all, exactly the same risk you already saw with enforceSchema in module 1.

Worked example: three reads, three schemas

Step 1 — The three files

Besides the seven orders_2026-08-*.csv files you already have from module 1, save these two new files in the same folder:

# dim_store.csv
store_id,store_name,city
S01,Kiosko Centro,Bogota
S02,Kiosko Norte,Lima
S03,Kiosko Sur,Santiago

# dim_product.csv
product_id,product_name,category,unit_cost
P001,Bottled Water 600ml,beverages,0.40
P002,Energy Bar,snacks,0.60
P003,Instant Coffee Sachet,beverages,0.35
P004,Phone Charger Cable,electronics,2.10

This is the same master data for Kiosko — three stores, four products — you already know from data-engineering-foundations-guide, and that data-modeling-for-analytics-guide turned into dim_store and dim_product with a surrogate key. Notice something important: unit_cost (here, in dim_product) isn't the same as unit_price (in orders) — unit_cost is what the product costs Kiosko, unit_price is what it charges the customer. This guide doesn't calculate margins with that difference again until module 7; for now, it's enough not to confuse the two columns.

Step 2 — Three StructType, three reads

# read_star_inputs.py
from pyspark.sql import SparkSession
from pyspark.sql.types import (
    StructType, StructField, StringType, IntegerType, DoubleType, TimestampType,
)

spark = SparkSession.builder.appName("kiosko-spark").master("local[*]").getOrCreate()

orders_schema = StructType([
    StructField("order_id", StringType(), False),
    StructField("store_id", StringType(), False),
    StructField("product_id", StringType(), False),
    StructField("quantity", IntegerType(), False),
    StructField("unit_price", DoubleType(), False),
    StructField("order_ts", TimestampType(), False),
])

dim_store_schema = StructType([
    StructField("store_id", StringType(), False),
    StructField("store_name", StringType(), False),
    StructField("city", StringType(), False),
])

dim_product_schema = StructType([
    StructField("product_id", StringType(), False),
    StructField("product_name", StringType(), False),
    StructField("category", StringType(), False),
    StructField("unit_cost", DoubleType(), False),
])

orders_df = spark.read.csv(
    "orders_2026-08-*.csv", schema=orders_schema, header=True, enforceSchema=False,
)
dim_store_df = spark.read.csv(
    "dim_store.csv", schema=dim_store_schema, header=True, enforceSchema=False,
)
dim_product_df = spark.read.csv(
    "dim_product.csv", schema=dim_product_schema, header=True, enforceSchema=False,
)

print(f"orders_df.count() = {orders_df.count()}")
print(f"dim_store_df.count() = {dim_store_df.count()}")
print(f"dim_product_df.count() = {dim_product_df.count()}")

print("\ndim_store_df, sorted by store_id:")
dim_store_df.orderBy("store_id").show(truncate=False)

print("dim_product_df, sorted by product_id:")
dim_product_df.orderBy("product_id").show(truncate=False)

spark.stop()

What to expect. Running python3 read_star_inputs.py, the output is exactly this (executed in this run, PySpark 4.2.0):

orders_df.count() = 40
dim_store_df.count() = 3
dim_product_df.count() = 4

dim_store_df, sorted by store_id:
+--------+-------------+--------+
|store_id|store_name   |city    |
+--------+-------------+--------+
|S01     |Kiosko Centro|Bogota  |
|S02     |Kiosko Norte |Lima    |
|S03     |Kiosko Sur   |Santiago|
+--------+-------------+--------+

dim_product_df, sorted by product_id:
+----------+---------------------+-----------+---------+
|product_id|product_name         |category   |unit_cost|
+----------+---------------------+-----------+---------+
|P001      |Bottled Water 600ml  |beverages  |0.4      |
|P002      |Energy Bar           |snacks     |0.6      |
|P003      |Instant Coffee Sachet|beverages  |0.35     |
|P004      |Phone Charger Cable  |electronics|2.1      |
+----------+---------------------+-----------+---------+

Three numbers confirmed at once: forty orders (the number you already know), three stores, four products. None of this joins anything yet — they're three completely independent DataFrame objects, each with its own read, its own schema, and — notice — unit_cost shows as 0.4, not 0.40, exactly the same display behavior you already saw with unit_price in module 1: Spark stores the correct numeric value, printSchema()/.show() just display it without extra trailing zeros.

Diagram: three shelves, three independent reads

flowchart TD
    subgraph Orders["Shelf 1: orders (7 files)"]
        A["orders_schema:\norder_id, store_id, product_id,\nquantity, unit_price, order_ts"]
    end
    subgraph Store["Shelf 2: dim_store (1 file)"]
        B["dim_store_schema:\nstore_id, store_name, city"]
    end
    subgraph Product["Shelf 3: dim_product (1 file)"]
        C["dim_product_schema:\nproduct_id, product_name,\ncategory, unit_cost"]
    end

    A --> D["orders_df\n40 rows"]
    B --> E["dim_store_df\n3 rows"]
    C --> F["dim_product_df\n4 rows"]

    D -.->|"still not joined"| G(("Lesson 3"))
    E -.->|"still not joined"| G
    F -.->|"still not joined"| G

Going deeper: why three separate StructType, not just one

You could, in theory, write a single giant StructType with the thirteen combined columns from the three tables and try to read all three files with it — but that wouldn't work, and it's worth understanding why, not just taking it on faith. Every call to spark.read.csv(schema=..., header=True) applies one schema to one file (or a set of files sharing the same format, like orders's seven). The schema tells Spark, column by column, in what order and with what type to expect that specific file's data — it makes no sense to ask Spark to apply a thirteen-column schema to a file that only has three, or the other way around.

This has a practical consequence you're going to use throughout this guide: every Kiosko table — orders, dim_store, dim_product, and later fact_orders_at_scale — has its own StructType, declared once, and that StructType is, in a real sense, that table's contract: anyone reading the code knows, without opening a single CSV file, exactly which columns and which types to expect. This discipline — an explicit schema per table, never inferred — is the same one you already saw in module 1, now applied to three tables instead of one.

Common mistakes

Copying orders_schema and only changing the variable name, without changing the fields. What happens: someone, in a hurry, copies and pastes orders_schema's StructType to create dim_store_schema, and only changes the variable name — accidentally leaving orders's six fields instead of dim_store's real three fields. Why it happens: copy-pasting is faster than writing from scratch, and the mistake doesn't show up until the code runs. How to spot it: if spark.read.csv("dim_store.csv", schema=dim_store_schema, ...) fails with an enforceSchema error similar to what you already saw in module 1 (CSV header does not conform to the schema), the schema almost certainly has the wrong number of fields, or the wrong fields. How to fix it: every new table deserves its own StructType, written by looking at that table's real columns — three for dim_store, four for dim_product — not an unreviewed copy of another table's schema.

Confusing dim_product's unit_cost with orders's unit_price. What happens: someone, when writing lesson 3's .join() or lesson 4's revenue calculation, uses unit_cost where they should use unit_price, or vice versa — both are DoubleType columns with similar names, in different tables. Why it happens: the two names are almost identical, and in English the difference between "cost" (what it costs Kiosko) and "price" (what Kiosko charges) is subtle if you don't read carefully. How to spot it: if your calculated revenue comes out much lower than expected (for example, if you accidentally calculate quantity * unit_cost instead of quantity * unit_price), suspect this mix-up first — it's exactly the mistake data-engineering-foundations-guide already documented as a common error in its own derived-columns module. How to fix it: revenue is always calculated with unit_price (from orders), never with unit_cost (from dim_product); unit_cost doesn't show up again in this guide until module 7, when you calculate margin (unit_price - unit_cost) to classify margin_category.

Assuming dim_store_df.count() or dim_product_df.count() are going to be large. What happens: someone, used to orders_df.count() being 40, is surprised to see dim_store_df.count() == 3 and dim_product_df.count() == 4, and wonders if something went wrong in the read. Why it happens: it's easy to forget that a star schema's dimension tables are, almost always, much smaller than the fact table — that's part of their definition, not a bug. How to spot it: check Kiosko's master data: three stores, four products, are the real data, not a trimmed-down sample. How to fix it: nothing to fix — three and four are the correct counts, and it's precisely because they're so small that these two tables are perfect candidates for a broadcast join, module 5's central topic in this guide.

Exercises

Exercise 1 — Verify the three tables' types with printSchema(). Add orders_df.printSchema(), dim_store_df.printSchema(), and dim_product_df.printSchema() to this lesson's script, and confirm unit_cost shows up as double, not as string.

print("orders_df.printSchema():")
orders_df.printSchema()
print("dim_store_df.printSchema():")
dim_store_df.printSchema()
print("dim_product_df.printSchema():")
dim_product_df.printSchema()

dim_product_df.printSchema():
root
 |-- product_id: string (nullable = true)
 |-- product_name: string (nullable = true)
 |-- category: string (nullable = true)
 |-- unit_cost: double (nullable = true)

Exercise 2 — Count dim_product's distinct categories. Using .select("category").distinct().count(), confirm how many distinct product categories exist in Kiosko, without looking at the CSV file directly.

num_categories = dim_product_df.select("category").distinct().count()
print(f"Distinct categories: {num_categories}")
dim_product_df.select("category").distinct().orderBy("category").show()

Distinct categories: 3
+-----------+
|   category|
+-----------+
|  beverages|
|electronics|
|     snacks|
+-----------+

Exercise 3 — Explain, without code, why a wrong schema in dim_store would silently break lesson 3's JOIN, not just the read. In 2-3 sentences, explain what would happen if dim_store_schema declared store_id as IntegerType instead of StringType, even if reading dim_store.csv threw no error.

Summary and next step

In this lesson you read, for the first time in this guide, the three complete tables that make up Kiosko's star schema: orders (40 rows, already known), dim_store (3 rows), and dim_product (4 rows), each with its own explicit StructType. None of the three got joined with the others yet — they're three independent DataFrame objects, ready for lesson 3's .join().

Before moving on you should be able to: write dim_store_schema's three fields and dim_product_schema's four from memory; explain the difference between unit_cost (in dim_product) and unit_price (in orders); and explain why a wrong schema in a dimension table would silently break a JOIN, not fail with a loud error.

Lesson 3 takes these three pieces and joins them, for the first time, with the DataFrame API's .join() — the first real step toward fact_orders.

Resources

• Apache Spark — SQL Getting Started (the explicit-schema read pattern this lesson repeats three times). spark.apache.org/docs/latest/sql-getting-started.html.
• Apache Spark — CSV Files (the enforceSchema documentation and why this guide always disables it, already cited in module 1). spark.apache.org/docs/latest/sql-data-sources-csv.html.
• data-modeling-for-analytics-guide DESIGN doc — the source of dim_store and dim_product with their exact columns, already built with a surrogate key in that guide. src/guides/data-modeling-for-analytics-guide/DISENO.md