RDDs: the original abstraction, seen once and left behind

Description

Before the DataFrame API existed — the one you're going to use for the rest of this guide, without exception, starting in lesson 4 — Spark had only one way to describe distributed data: the RDD, Resilient Distributed Dataset. This lesson shows you what an RDD is, with real, executed code, and why this guide — following the market evidence cited in module 1 (Python 71% versus Scala 12%, and the explicit warning against a "curriculum heavy on RDDs") — uses it exactly once, here, for historical contrast, and never builds anything on it again.

Connection to the module. This lesson exists solely so lesson 4 has something real to compare against. You're not going to build any Kiosko pipeline with RDDs in this lesson or anywhere else in the guide — the goal is purely to understand the shape of the original API, so you can appreciate, with evidence, why the industry moved to something else.

An analogy: the blank notebook before the printed form

Picture a new employee at an office, in charge of writing down customer orders. If you give them a completely blank notebook, with no structure at all, they can write down whatever they want, however they want: one line per order, three lines, their own abbreviations, any format that occurs to them. It's flexible — they can write down absolutely anything — but nobody else (not even a very experienced supervisor) can look at that notebook and know, ahead of time, which fields exist, where the customer's name or the order amount is, without reading every line one by one.

An RDD is exactly that blank notebook: a distributed collection of arbitrary Python objects — they can be numbers, dictionaries, tuples, instances of custom classes — with no schema Spark can inspect ahead of time. Spark knows there's a collection of "things" spread across several partitions, but it doesn't know, without running your Python code line by line, what shape each "thing" has. The DataFrame API, which you see in lesson 4, is the printed form: boxes with names and types declared up front (order_id: string, quantity: integer), that anyone — including Spark's Catalyst optimizer — can read and reason about without running a single line of your code.

Worked example: your first (and only) RDD in this guide

# rdd_contrast.py
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("kiosko-spark").master("local[*]").getOrCreate()
sc = spark.sparkContext

numbers_rdd = sc.parallelize([1, 2, 3])
doubled_rdd = numbers_rdd.map(lambda x: x * 2)
result = doubled_rdd.collect()

print(f"result = {result}")
print(f"numbers_rdd.getNumPartitions() = {numbers_rdd.getNumPartitions()}")
print(f"type(doubled_rdd) = {type(doubled_rdd)}")

spark.stop()

What to expect. Running python3 rdd_contrast.py, the output is exactly this (executed in this run):

result = [2, 4, 6]
numbers_rdd.getNumPartitions() = 12
type(doubled_rdd) = <class 'pyspark.core.rdd.PipelinedRDD'>

Notice three things. First, sc.parallelize([1, 2, 3]) takes an ordinary Python collection — a list of three integers, living in the driver — and turns it into an RDD, conceptually distributing it across the available partitions. Second, numbers_rdd.getNumPartitions() gives 12 on this machine — the same number as sc.defaultParallelism from the previous lesson — even though the original list only had three elements: Spark reserves twelve potential partitions based on the default parallelism, though most end up empty with such a small input. Third, and most important: .map(lambda x: x * 2) is a transformation on the RDD — lazy, just like you're going to see with DataFrames in lesson 5 — and .collect() is the action that triggers it: only when .collect() is called does Spark actually multiply each number by two and bring the result back to the driver as a regular Python list ([2, 4, 6]).

Diagram: what each piece does

flowchart LR
    A["[1, 2, 3]\nregular Python list,\nlives in the driver"] -->|"sc.parallelize()"| B["numbers_rdd\ndistributed RDD,\nno known schema"]
    B -->|".map(lambda x: x * 2)\n(transformation, lazy)"| C["doubled_rdd\nPipelinedRDD,\nstill nothing computed"]
    C -->|".collect()\n(action, triggers the work)"| D["[2, 4, 6]\nregular Python list,\nback in the driver"]

Going deeper: what an RDD is (and is NOT), with the official quote

Spark's official documentation — the RDD Programming Guide, still maintained even though modern documentation's focus is on DataFrames — defines an RDD like this: "The main abstraction Spark provides is a resilient distributed dataset (RDD), which is a collection of elements partitioned across the nodes of the cluster that can be operated on in parallel." Three words in that definition deserve attention: "collection of elements" — with no mention at all of schema, column types, or field names. An RDD can literally contain any serializable Python object: numbers, tuples, dictionaries, dataclass instances, even other RDDs nested in theory. That total flexibility is also its central limitation: without knowing the shape of the data ahead of time, Spark can't optimize anything about it — it can only faithfully execute the exact sequence of Python operations you gave it, element by element.

The same official guide also confirms the transformations-and-actions pattern you already saw with .map() and .collect(), and which reappears identically — but with more information available to Catalyst — in lesson 4's DataFrame API: "RDDs support two types of operations: transformations, which create a new dataset from an existing one, and actions, which return a value to the driver program after running a computation on the dataset." This same vocabulary — transformation, action — is what you're going to use for the rest of this guide, applied to DataFrames instead of RDDs. It isn't a naming coincidence: it's Spark's same central idea, lazy evaluation included, inherited from the original API into the modern one.

Before SparkSession existed (the one you use throughout this guide), the way to start Spark was different — it's worth seeing once, even though you're never going to write it anywhere else in the guide, because you're likely to run into it if you ever read a Spark project from several years back:

# historical form, PRE-SparkSession (don't use this in this guide)
from pyspark import SparkConf, SparkContext

conf = SparkConf().setAppName("kiosko-spark").setMaster("local[*]")
sc = SparkContext(conf=conf)

SparkSession, the one you build throughout this guide with SparkSession.builder...getOrCreate(), is the modern unified interface that wraps both the SparkContext (the entry point to the RDD API, still accessible as spark.sparkContext) and the entry point to the DataFrame API and Spark SQL. That's exactly why, in this lesson's worked example, the RDD was reached through spark.sparkContext.parallelize(...) — the SparkContext still exists underneath, available when you genuinely need it (as in this one lesson), but it's no longer the main entry point of any script in this guide.

Common mistakes

Trying to use .select() or .filter(col(...) == ...) directly on an RDD. What happens: someone, used to the DataFrame syntax from the lessons that follow, tries to write numbers_rdd.select(...) or numbers_rdd.filter(col("x") > 1) on an RDD, and gets an AttributeError. Why it happens: .select() and the col() function are part of the DataFrame API, which depends on Spark knowing column names and types — something an RDD, by definition, doesn't have. How to spot it: the error message is going to say something like 'RDD' object has no attribute 'select' — a clear sign you're mixing the two APIs. How to fix it: an RDD only has the RDD API's methods (.map(), .filter() with a regular Python lambda, .reduce(), .collect()) — to use .select()/col(), you need a DataFrame, not an RDD. Lesson 4 shows how to convert one into the other when you genuinely need to (orders_df.rdd to go from DataFrame to RDD; spark.createDataFrame(rdd, schema) for the reverse path).

Thinking .explain() works the same on an RDD as on a DataFrame. What happens: someone, curious to see an RDD's execution plan (the way you do with DataFrames in lesson 6), tries calling numbers_rdd.explain() and gets an error. Why it happens: .explain() is a DataFrame API method that shows Catalyst's plan — and Catalyst has nothing to optimize on an RDD, because an RDD lacks the schema Catalyst needs to reason. How to spot it: the error literally says 'RDD' object has no attribute 'explain' — you're going to see this same evidence, verified with real code, in lesson 4. How to fix it: there's no way to ask an RDD for an optimized execution plan — this is, precisely, the central limitation that motivated creating the DataFrame API, and the point lesson 4 demonstrates with direct evidence.

Writing a full pipeline with RDDs "because you already learned it somewhere." What happens: someone with prior experience from an older course or book on Spark, which taught RDDs as the main interface, keeps writing .map()/.reduceByKey() instead of .groupBy().agg() when working with Kiosko in the modules that follow. Why it happens: educational material written before the DataFrame API matured — or that simply never got updated — still teaches RDDs as the main path, and that habit is hard to unlearn. How to spot it: if in module 3 (where fact_orders gets rebuilt) you find yourself writing .map() or .reduceByKey() instead of .join(), .groupBy(), and .agg(), you're using the wrong API for this guide. How to fix it: starting with lesson 4, this entire guide — with no exception, including the module 8 capstone — uses exclusively the DataFrame API. If you ever need RDDs in a real project outside this guide (increasingly rare, per module 1's market evidence), this lesson gives you the minimum vocabulary to recognize them, not to build on them by default.

Exercises

Exercise 1 — Repeat the example with a different transformation. Using sc.parallelize(), create an RDD with the numbers 1 through 10 and use .filter() (not .map()) to keep only the even numbers, before a final .collect().

numbers_rdd = sc.parallelize(range(1, 11))
even_rdd = numbers_rdd.filter(lambda x: x % 2 == 0)
result = even_rdd.collect()
print(f"result = {result}")

result = [2, 4, 6, 8, 10]

Exercise 2 — Chain two transformations before the .collect(). Extend exercise 1: after filtering the even numbers, add 100 to each one with .map(), and confirm with a print that neither transformation prints anything on its own (only the final .collect() produces a result).

numbers_rdd = sc.parallelize(range(1, 11))
even_rdd = numbers_rdd.filter(lambda x: x % 2 == 0)
print("Right after .filter(): nothing to print, just an uncomputed RDD")

shifted_rdd = even_rdd.map(lambda x: x + 100)
print("Right after .map(): still nothing -- still a plan, not a result")

result = shifted_rdd.collect()
print(f"Final result, only after .collect(): {result}")

Right after .filter(): nothing to print, just an uncomputed RDD
Right after .map(): still nothing -- still a plan, not a result
Final result, only after .collect(): [102, 104, 106, 108, 110]

Exercise 3 — Explain, without code, what information Spark is missing when it works with an RDD. In 2-3 sentences, and using this lesson's official quote from the "going deeper" section, explain what concrete information Spark has about a DataFrame that it does not have about an RDD, and why that difference matters for optimization.

Summary and next step

In this lesson you saw, for the first and last time in this guide, Spark's original API: RDDs. You confirmed, with executed code, that sc.parallelize()/.map()/.collect() follow the same pattern of lazy transformations and actions that dominates the rest of this guide — just without any known schema, which keeps Spark from optimizing anything about them. You also saw the historical way of starting Spark (SparkConf/SparkContext, before SparkSession) in case you ever run into it in older code.

Before moving on you should be able to: write sc.parallelize([...]).map(...).collect() from memory; explain why an RDD has no .explain(); and explain, in your own words, what structural information an RDD is missing that a DataFrame has.

Lesson 4 takes this same limitation — no schema, no possible optimization — and contrasts it, with real code executed side by side, against the DataFrame API: the interface you're going to use for the rest of this guide, without exception.

Resources

• Apache Spark — RDD Programming Guide (official RDD definition: "a collection of elements partitioned across the nodes of the cluster that can be operated on in parallel", and of transformations/actions). spark.apache.org/docs/latest/rdd-programming-guide.html.
• Apache Spark — SQL Programming Guide (context for why the DataFrame API emerged as a layer over RDDs, with "richer optimizations" — covered in depth in lesson 4). spark.apache.org/docs/latest/sql-programming-guide.html.
• This guide's DESIGN doc (spark-and-distributed-processing-guide/DISENO.md) — the explicit boundary: "RDDs are named exactly once, for historical contrast (M2), and no pipeline is ever built on them."