What a partition actually is

Description

orders_df.count() gave you 40 back in module 1, and you never asked how Spark arrived at that number internally — you didn't need to know. This lesson opens that box: a Spark DataFrame doesn't live as a single block of data in one place, it lives spread across partitions, and each partition is the real unit of work an executor processes. You're going to measure, with Spark's own API, how many partitions orders_df has today, how many rows fall into each one, and what decides that number.

Connection to the module. This is the module's first technical piece, the one lesson 1 mapped out. Without understanding what a partition is, lesson 3's word "shuffle" has nothing to stand on — a shuffle is, precisely, a reorganization of partitions, so first you need to know what's being reorganized.

An analogy: the order's boxes, before anyone moves them

Pick back up lesson 1's analogy: a large order, split into boxes, spread across several trucks. Before asking what happens when someone asks for the boxes to be reorganized by a new criterion (that's lesson 3), it's worth asking something simpler: who decided, in the first place, how many boxes to make and what goes in each one? You don't make that decision directly when you write spark.read.csv(...) — Spark makes it, based on the size of the source files and how many "trucks" (available CPU cores) your machine has to work in parallel. This lesson measures that decision with real evidence, over the same seven Kiosko files you already know.

Worked example: counting orders_df's partitions

# partition_basics.py
import glob

from pyspark.sql import SparkSession
from pyspark.sql.types import (
    StructType, StructField, StringType, IntegerType, DoubleType, TimestampType,
)

spark = (
    SparkSession.builder
    .appName("kiosko-spark")
    .master("local[*]")
    .getOrCreate()
)
print(f"Spark version: {spark.version}")
print(f"sparkContext.defaultParallelism = {spark.sparkContext.defaultParallelism}\n")

orders_schema = StructType([
    StructField("order_id", StringType(), False),
    StructField("store_id", StringType(), False),
    StructField("product_id", StringType(), False),
    StructField("quantity", IntegerType(), False),
    StructField("unit_price", DoubleType(), False),
    StructField("order_ts", TimestampType(), False),
])
orders_df = spark.read.csv(
    sorted(glob.glob("orders_2026-08-*.csv")), schema=orders_schema, header=True, enforceSchema=False,
)

print(f"orders_df.count() = {orders_df.count()}")
print(f"orders_df.rdd.getNumPartitions() = {orders_df.rdd.getNumPartitions()}")

# How many rows landed in each partition -- mapPartitions runs a function
# per partition, without crossing data between them.
def count_rows_per_partition(rows):
    yield sum(1 for _ in rows)

rows_per_partition = orders_df.rdd.mapPartitions(count_rows_per_partition).collect()
print(f"Rows per partition: {rows_per_partition}")
print(f"Sum across all partitions: {sum(rows_per_partition)}")

spark.stop()

What to expect. Running python3 partition_basics.py, the output is exactly this (executed in this run, PySpark 4.2.0, on a machine with 12 logical cores):

Spark version: 4.2.0
sparkContext.defaultParallelism = 12

orders_df.count() = 40
orders_df.rdd.getNumPartitions() = 7

Rows per partition: [9, 8, 7, 6, 5, 3, 2]
Sum across all partitions: 40

Two numbers deserve immediate attention. First, orders_df.rdd.getNumPartitions() gives 7 — not 12 (this machine's core count), not 1 (a single block). It's exactly the number of CSV files you read: orders_2026-08-03.csv through orders_2026-08-09.csv, seven files. When Spark reads multiple small files, by default it assigns each one its own partition — each truck takes one whole file. Second, the list [9, 8, 7, 6, 5, 3, 2] sums to exactly 40, confirming no row got lost or duplicated while being spread out — these are the same per-file counts you already saw in module 1 (8, 6, 2, 5, 7, 9, 3), just in a different order, because the order Spark assigns partitions to files doesn't necessarily follow the glob's alphabetical order.

Diagram: seven files, seven partitions, one single executor

flowchart TD
    subgraph Origen["Seven CSV files on disk"]
        F1["orders_2026-08-03.csv\n8 rows"]
        F2["orders_2026-08-04.csv\n6 rows"]
        F3["orders_2026-08-05.csv\n2 rows"]
        F4["orders_2026-08-06.csv\n5 rows"]
        F5["orders_2026-08-07.csv\n7 rows"]
        F6["orders_2026-08-08.csv\n9 rows"]
        F7["orders_2026-08-09.csv\n3 rows"]
    end

    subgraph Particiones["orders_df -- 7 partitions"]
        P1["Partition 0"]
        P2["Partition 1"]
        P3["Partition 2"]
        P4["Partition 3"]
        P5["Partition 4"]
        P6["Partition 5"]
        P7["Partition 6"]
    end

    F1 --> P1
    F2 --> P2
    F3 --> P3
    F4 --> P4
    F5 --> P5
    F6 --> P6
    F7 --> P7

    subgraph Ejecucion["local[*] -- 12 cores available"]
        E["A single JVM process,\nup to 12 tasks in parallel"]
    end

    Particiones --> E

Notice something important in this diagram: even though the machine has 12 available cores (defaultParallelism = 12), orders_df only has 7 partitions — Spark doesn't invent partitions that don't correspond to any real data source. With only seven partitions, five of this machine's cores would sit idle if you tried to process orders_df at maximum parallelism — a detail worth remembering when lesson 6 talks about repartition().

Going deeper: who decides the partition count, and by what rules

For a small file like each of Kiosko's seven (under half a kilobyte), the rule is simple: one partition per file. But that rule isn't universal — for a single large file, Spark uses different logic, governed mainly by an officially documented setting:

You can confirm the current value in your own session:

print(spark.conf.get("spark.sql.files.maxPartitionBytes"))
# 134217728b

The intuition is this: if a source file weighs, say, 600 MB, and the per-partition limit is 128 MB, Spark needs at least five partitions (600 / 128 ≈ 4.7, rounded up) so none of them exceeds that limit. But that calculation also has a floor: Spark usually doesn't generate fewer partitions than the cluster's available parallelism (defaultParallelism), because doing so would leave cores idle from the very start — exactly what lesson 4's example avoids, where you're going to see a single file over half a gigabyte end up with a partition count that matches, not by coincidence, the number of cores on the machine processing it.

This behavior — one partition per small file, or a number calculated from total size and available parallelism for a large file — is exactly what you're going to contrast in lesson 4: Kiosko's seven tiny files produce 7 partitions; a single ten-million-row file is going to produce a very different number, calculated with this same mechanical logic, not picked by hand.

Common mistakes

Assuming the partition count equals the machine's core count. What happens: someone sees defaultParallelism = 12 and expects orders_df.rdd.getNumPartitions() to also give 12, and is surprised to see 7. Why it happens: it's tempting to think Spark always "uses all available cores" to decide how many partitions to create, as if cluster parallelism were the only factor. How to spot it: if your partition count doesn't match defaultParallelism, it isn't a bug — first check how many source files you're reading (for small files, it's usually one partition per file) or how large the single file you're reading is, against spark.sql.files.maxPartitionBytes. How to fix it: never assume the partition count — always measure it with .rdd.getNumPartitions(), as this lesson's worked example does, before reasoning about parallelism or cost.

Confusing "more partitions" with "always better." What happens: someone, learning that partitions are the unit of parallelism, assumes more partitions always means more speed, and tries to maximize the partition count of any DataFrame. Why it happens: if parallelism helps, it seems reasonable that "more parallelism" helps more. How to spot it: with only 7 partitions over 40 rows, you already have more partitions than useful rows per partition to make distributing anything worthwhile — splitting those forty rows into, say, two hundred partitions wouldn't speed anything up, because each partition would end up with a fraction of a row or none, and the overhead of coordinating two hundred tasks would outweigh any benefit. How to fix it: the right number of partitions depends on the real data volume and the available parallelism, not a fixed number — module 4's lesson 6 covers this exact decision in depth with repartition() and coalesce().

Reading .rdd.getNumPartitions() as a free operation in any context. What happens: someone calls .rdd.getNumPartitions() repeatedly inside a production pipeline, assuming it's as cheap as reading a Python attribute. Why it happens: the method's name doesn't suggest any execution cost. How to spot it: on a DataFrame that hasn't materialized yet (for example, with Adaptive Query Execution active and a plan depending on runtime statistics), accessing .rdd can force Spark to turn the logical plan into a concrete RDD, which in some cases triggers real work — a detail you're going to see with evidence in lesson 7, when you compare partitions before and after an action with AQE active. How to fix it: in this lesson's code, calling .rdd.getNumPartitions() on a DataFrame freshly read from a file (with no intermediate transformations depending on runtime statistics) is cheap and safe; in more complex plans, later in this guide, you're going to learn to tell when that same call can trigger extra work.

Exercises

Exercise 1 — Confirm orders_df's partitions don't overlap. Using mapPartitions, instead of just counting rows per partition, collect the set of order_id values that landed in each partition, and confirm no order_id shows up in two different partitions.

def order_ids_per_partition(rows):
    yield [r["order_id"] for r in rows]

ids_per_partition = orders_df.rdd.mapPartitions(order_ids_per_partition).collect()
all_ids = [oid for partition in ids_per_partition for oid in partition]
unique_ids = set(all_ids)

print(f"Total order_id seen (with repeats): {len(all_ids)}")
print(f"Total unique order_id: {len(unique_ids)}")
assert len(all_ids) == len(unique_ids) == 40
print("Verification: no order_id shows up in more than one partition -> OK")

Total order_id seen (with repeats): 40
Total unique order_id: 40
Verification: no order_id shows up in more than one partition -> OK

Exercise 2 — Predict the partition count if you read only one file. Without running anything yet, predict how many partitions a DataFrame read with spark.read.csv("orders_2026-08-03.csv", ...) would have — a single file, not all seven. Then, verify it.

single_file_df = spark.read.csv("orders_2026-08-03.csv", schema=orders_schema, header=True, enforceSchema=False)
print(f"single_file_df.rdd.getNumPartitions() = {single_file_df.rdd.getNumPartitions()}")
print(f"single_file_df.count() = {single_file_df.count()}")

single_file_df.rdd.getNumPartitions() = 1
single_file_df.count() = 8

Exercise 3 — Explain, without code, why [9, 8, 7, 6, 5, 3, 2]'s order doesn't match the files' order by date. In 2-3 sentences, explain why the worked example's list of rows per partition doesn't show up in the same order as the alphabetically sorted files (8, 6, 2, 5, 7, 9, 3).

Summary and next step

In this lesson you measured, with real evidence, what a partition is: the unit of work Spark assigns to an executor. You confirmed orders_df — Kiosko's seven real files — has 7 partitions, one per file, with [9, 8, 7, 6, 5, 3, 2] rows spread across them, summing to the usual 40. You also saw the rule governing that number for larger files: spark.sql.files.maxPartitionBytes (128 MB by default), further bounded by the machine's available parallelism.

Before moving on you should be able to: explain what a partition is without using the word "truck"; predict how many partitions a DataFrame read from a single small file would have; and explain why the partition count isn't, in general, equal to defaultParallelism.

Now that you know what a partition is, lesson 3 answers the question lesson 1's analogy left open: which operations force Spark to move rows from one partition to another, and why does that carry a cost no other operation has?

Resources

• Apache Spark — SQL Performance Tuning, spark.sql.files.maxPartitionBytes properties table (the exact default value, 134217728, and its official description — the foundation for this lesson's "going deeper" section). spark.apache.org/docs/latest/sql-performance-tuning.html.
• Apache Spark — RDD Programming Guide, "RDD Operations" section (the definition of mapPartitions as a transformation operating partition by partition, used in this lesson's worked example). spark.apache.org/docs/latest/rdd-programming-guide.html.
• This guide's DESIGN doc (spark-and-distributed-processing-guide/DISENO.md) — this module's full objective within the eight-module plan.