The DataFrame API as the modern interface

Description

The previous lesson showed you an RDD and told you, in prose, that Spark can't optimize anything about it because it doesn't know its schema. This lesson proves it with real, executed code, side by side: the same filter over the same Kiosko data, once with the RDD API and once with the DataFrame API, showing exactly what information Spark gains — and what Catalyst, the optimizer, gains — when you use the modern interface. Starting with this lesson, the entire guide uses exclusively the DataFrame API, with no exception at all.

Connection to the module. This lesson closes out the RDD-versus-DataFrame contrast lesson 3 opened, and establishes, for good, what the vehicle for all the work that follows is going to be: in lessons 5 through 8 of this module, and in the six remaining modules of this guide, every orders_df, every .filter(), every .groupBy() is DataFrame API, never RDD.

An analogy: the spreadsheet with named columns

Pick back up with lesson 3's new employee, who wrote down orders in a blank notebook, with no structure at all. Now imagine that, instead of that notebook, they're given a spreadsheet with columns already named and with an expected data type in each one: "Customer name" (text), "Quantity" (integer), "Unit price" (decimal number). The employee still fills in rows, one per order — the real work didn't change — but now an accountant, without reading a single row, can look at the column headers and know ahead of time which operations are valid: they can sum the "Quantity" column because they know it's numeric, they can flag it if someone tries to write text into the price column, they can reorganize the order in which columns get processed to make the work more efficient — all of that, without having read the real content of a single row.

That's exactly what Spark gains with a DataFrame versus an RDD: columns with names and types declared up front, which the Catalyst optimizer — the "accountant" in this analogy — can inspect and use to make decisions, without running a single line of your Python code. Spark's official documentation puts it with a comparison almost identical to this analogy: a DataFrame is "conceptually equivalent to a table in a relational database" — with named, typed columns from the start.

Worked example: the same filter, two APIs, a direct contrast

# dataframe_vs_rdd.py
from pyspark.sql import SparkSession
from pyspark.sql.types import (
    StructType, StructField, StringType, IntegerType, DoubleType, TimestampType,
)
from pyspark.sql.functions import col

spark = SparkSession.builder.appName("kiosko-spark").master("local[*]").getOrCreate()

orders_schema = StructType([
    StructField("order_id", StringType(), False),
    StructField("store_id", StringType(), False),
    StructField("product_id", StringType(), False),
    StructField("quantity", IntegerType(), False),
    StructField("unit_price", DoubleType(), False),
    StructField("order_ts", TimestampType(), False),
])
orders_df = spark.read.csv(
    "orders_2026-08-*.csv", schema=orders_schema, header=True, enforceSchema=False,
)

# Via RDD: no schema, rows as Python Row objects, no plan for Catalyst to optimize
orders_rdd = orders_df.rdd
s01_via_rdd = orders_rdd.filter(lambda row: row.store_id == "S01" and row.quantity > 1)
print("Via RDD -- filter with a Python lambda:")
print(f"  s01_via_rdd.count() = {s01_via_rdd.count()}")
print(f"  first row: {s01_via_rdd.first()}")
try:
    s01_via_rdd.explain()
except AttributeError as e:
    print(f"  s01_via_rdd.explain() -> AttributeError: {e}")

print()

# Via DataFrame API: Catalyst knows the schema and the plan
s01_via_df = orders_df.filter(col("store_id") == "S01").filter(col("quantity") > 1)
print("Via DataFrame API -- filter with a Column:")
print(f"  s01_via_df.count() = {s01_via_df.count()}")
print("  s01_via_df.explain():")
s01_via_df.explain()

spark.stop()

What to expect. Running python3 dataframe_vs_rdd.py, the output is exactly this (executed in this run, over module 1's same forty Kiosko rows):

Via RDD -- filter with a Python lambda:
  s01_via_rdd.count() = 10
  first row: Row(order_id='ORD-6001', store_id='S01', product_id='P001', quantity=6, unit_price=0.55, order_ts=datetime.datetime(2026, 8, 8, 8, 0))
  s01_via_rdd.explain() -> AttributeError: 'PipelinedRDD' object has no attribute 'explain'

Via DataFrame API -- filter with a Column:
  s01_via_df.count() = 10
  s01_via_df.explain():
== Physical Plan ==
*(1) Filter (((isnotnull(store_id#1) AND isnotnull(quantity#3)) AND (store_id#1 = S01)) AND (quantity#3 > 1))
+- FileScan csv [order_id#0,store_id#1,product_id#2,quantity#3,unit_price#4,order_ts#5] Batched: false, DataFilters: [isnotnull(store_id#1), isnotnull(quantity#3), (store_id#1 = S01), (quantity#3 > 1)], Format: CSV, Location: InMemoryFileIndex(7 paths)[file:/Users/your-username/kiosko-spark/orders_2026-08-03.csv, ...], PartitionFilters: [], PushedFilters: [IsNotNull(store_id), IsNotNull(quantity), EqualTo(store_id,S01), GreaterThan(quantity,1)], ReadSchema: struct<order_id:string,store_id:string,product_id:string,quantity:int,unit_price:double,order_ts:...

(The Location field in this plan always shows the real absolute path of the folder where you ran the script — the one above is an illustrative form; on your own machine it's going to show your own working directory, truncated by Spark the same way. Lesson 6 shows this same field without truncation, with explain(mode="formatted"), and there the full path captured from a real run is preserved.)

Four observations, in order of importance. First, both paths give the same correct result — count() = 10 — confirming that the choice of API isn't a matter of correctness, it's a matter of how much information Spark has available to work better. Second, orders_df.rdd is how you "drop down" from DataFrame to RDD when you genuinely need to — each row turns into a regular Python Row object, with attribute access (row.store_id), but with no declared type Spark can inspect without running your lambda. Third, and this lesson's central finding: s01_via_rdd.explain() throws a real AttributeError — RDDs have no execution plan to show, because there's nothing Catalyst has optimized; that method simply doesn't exist on an RDD object. Fourth, s01_via_df.explain() does produce a full, readable physical plan: notice PushedFilters: [IsNotNull(store_id), IsNotNull(quantity), EqualTo(store_id,S01), GreaterThan(quantity,1)] — Catalyst took the two chained .filter() calls and pushed them all the way down to the CSV file scan itself, something that's only possible because Catalyst knows, ahead of time, that store_id and quantity are real columns with known types.

Diagram: the same question, two paths with different information

flowchart TD
    subgraph RDD["RDD path"]
        A["orders_df.rdd"] --> B["Row objects,\nno declared types\nfor Catalyst"]
        B --> C[".filter(lambda row: ...)\nopaque Python function --\nCatalyst can't look inside"]
        C --> D["count() = 10\ncorrect, but no plan,\nno .explain() possible"]
    end

    subgraph DF["DataFrame API path"]
        E["orders_df"] --> F["known schema:\nstore_id: string,\nquantity: integer"]
        F --> G[".filter(col(...) == ...)\nColumn expression --\nCatalyst CAN look inside"]
        G --> H["count() = 10\nsame result,\nWITH optimized plan\nand PushedFilters"]
    end

Going deeper: why "knowing the schema" changes everything, per the documentation itself

Spark's SQL Programming Guide explains, plainly, why this difference exists: "Unlike the basic Spark RDD API, the interfaces provided by Spark SQL provide Spark with more information about the structure of both the data and the computation being performed. Internally, Spark SQL uses this extra information to perform extra optimizations." The same guide precisely defines DataFrame: "A DataFrame is a Dataset organized into named columns. It is conceptually equivalent to a table in a relational database or a data frame in R/Python, but with richer optimizations under the hood."

"Richer optimizations" isn't an empty marketing phrase — this lesson's worked example makes it concrete: the PushedFilters you saw in the physical plan means Spark, before reading a single byte of the CSV into memory, already knows it only needs rows where store_id = 'S01' and quantity > 1, and can apply that filter during the read, not after. With the RDD path, by contrast, Spark has to read each full row, convert it into a Row object, and only then run your Python lambda over it to decide whether to keep it — there's no way to skip that work ahead of time, because the condition lives inside arbitrary Python code Catalyst can't inspect without running it.

This difference becomes much more important as volume grows — exactly module 4's topic in this guide, with the ten-million-row synthetic dataset: over forty rows, the performance difference between the two paths is negligible; over ten million, a filter that can be pushed down to the data source (PushedFilters) avoids moving a huge number of rows into memory that were going to get discarded anyway. This guide doesn't measure that difference with a stopwatch — by design, per this guide's rules — but this same lesson's .explain() plan is already evidence of why that difference would exist, with no need to measure laptop times that aren't reproducible across machines.

Common mistakes

Converting to RDD "just to inspect," without realizing the cost of lost information. What happens: someone, familiar with pure Python's .map()/.filter() syntax, converts a DataFrame to an RDD with .rdd to do a simple transformation the DataFrame API already supports directly (as this worked example does, deliberately, for the demonstration). Why it happens: for someone who just learned RDDs in lesson 3, .map(lambda row: ...) syntax can feel more familiar than col(...), especially at first. How to spot it: if in the rest of this guide — starting with this lesson — you find yourself writing .rdd.map(...) instead of .select()/.filter()/.withColumn() directly on the DataFrame, you lost, with no need to, all the schema information Catalyst could have used. How to fix it: unless you have a very specific reason (legacy PySpark code already written with RDDs, or an operation genuinely impossible to express with the DataFrame API — increasingly rare), always stay in the DataFrame API. This worked example converts to RDD solely for pedagogical contrast, not as a pattern to repeat.

Assuming .explain() is only for debugging errors. What happens: someone sees .explain() as a tool used only when something went wrong, and never runs it on code that "already works." Why it happens: in many programming languages, inspecting an operation's "internal plan" feels like an advanced debugging activity, not everyday work. How to spot it: if you never called .explain() on a DataFrame before running it at scale, you don't know whether Spark is applying PushedFilters, or how expensive the real plan is. How to fix it: as you saw in this lesson, .explain() executes nothing — it doesn't trigger any job (you're going to verify this with evidence in lesson 6) — and is practically free to call. Turning it into a habit, even on code that "already works," is exactly the discipline module 6 of this guide (Catalyst, .explain(), and caching) builds in depth.

Thinking the DataFrame API is "less powerful" than RDD because it has less flexibility for arbitrary Python types. What happens: someone notices an RDD can hold any Python object — an instance of a custom class, an arbitrary nested structure — while a DataFrame is limited to Spark SQL types (StringType, IntegerType, structs, arrays), and concludes the DataFrame API is "less capable." Why it happens: more type flexibility intuitively sounds like "more power." How to spot it: if your reason for preferring RDDs is "I need to store arbitrary Python objects in each row," it's worth checking whether that design genuinely needs it, or whether it can be expressed with Spark SQL's structured types (including nested StructType, ArrayType, MapType — which cover the vast majority of real cases). How to fix it: an RDD's total flexibility has a real, measurable cost in lost optimization, as this very lesson's .explain() showed. Module 1's market evidence (Python at 71% of postings, and the warning against a "curriculum heavy on RDDs") confirms that, in industry practice, that flexibility rarely justifies that cost.

Exercises

Exercise 1 — Reproduce the contrast with a different condition. Repeat this lesson's worked example, but filtering by product_id == "P001" instead of store_id == "S01" with quantity > 1. Confirm both paths (RDD and DataFrame) give the same count, and that only the DataFrame path produces a plan with .explain().

p001_via_rdd = orders_df.rdd.filter(lambda row: row.product_id == "P001")
print(f"Via RDD: {p001_via_rdd.count()}")

p001_via_df = orders_df.filter(col("product_id") == "P001")
print(f"Via DataFrame: {p001_via_df.count()}")
p001_via_df.explain()

Via RDD: 16
Via DataFrame: 16
== Physical Plan ==
*(1) Filter (isnotnull(product_id#2) AND (product_id#2 = P001))
+- FileScan csv [...] PushedFilters: [IsNotNull(product_id), EqualTo(product_id,P001)], ...

Exercise 2 — Confirm orders_df.rdd keeps the correct data, it only loses the explicit schema. Using orders_df.rdd.first(), print the first row as a Row object, then use .asDict() on that same row to turn it into a regular Python dictionary.

first_row = orders_df.rdd.first()
print(f"As Row: {first_row}")
print(f"As dict: {first_row.asDict()}")

As Row: Row(order_id='ORD-6001', store_id='S01', product_id='P001', quantity=6, unit_price=0.55, order_ts=datetime.datetime(2026, 8, 8, 8, 0))
As dict: {'order_id': 'ORD-6001', 'store_id': 'S01', 'product_id': 'P001', 'quantity': 6, 'unit_price': 0.55, 'order_ts': datetime.datetime(2026, 8, 8, 8, 0)}

Exercise 3 — Explain, without code, what PushedFilters means in your own words. Using this lesson's worked example .explain() plan, explain in 2-3 sentences what it means for a filter to show up in PushedFilters, and why that's only possible with the DataFrame API, not with RDDs.

Summary and next step

In this lesson you saw, with executed code and not just documentation's word, the real difference between RDD and the DataFrame API: the same filter, the same correct result (10 orders from S01 with quantity > 1), but only the DataFrame path produced an inspectable execution plan with .explain() — the RDD path literally threw an AttributeError, because there's nothing Catalyst can optimize over data with no known schema. From here on, this entire guide uses the DataFrame API, without exception.

Before moving on you should be able to: explain why .explain() doesn't exist on an RDD; quote, in your own words, the SQL Programming Guide's official phrase about "richer optimizations"; and recognize PushedFilters in an execution plan as evidence Catalyst moved a filter down to the data source.

With RDD and the DataFrame API now compared, lesson 5 digs into the behavior both share and that matters most in this whole module: lazy evaluation — why no transformation, on either API, executes anything until an action arrives.

Resources

• Apache Spark — SQL Programming Guide (the exact DataFrame vs RDD comparison: "more information about the structure... extra optimizations", and the definition of DataFrame as "conceptually equivalent to a table in a relational database... but with richer optimizations under the hood"). spark.apache.org/docs/latest/sql-programming-guide.html.
• Apache Spark — RDD Programming Guide (reference for the RDD API used in this lesson's contrast: .rdd, .map(), .filter() with Python functions). spark.apache.org/docs/latest/rdd-programming-guide.html.
• This guide's DESIGN doc (spark-and-distributed-processing-guide/DISENO.md) — the market evidence (Python 71% versus Scala 12%, warning against a curriculum heavy on RDDs) that justifies this design decision.