When caching only costs memory

Description

Lesson 5 demonstrated the case where .cache() helps: three different queries reusing the same DataFrame, and a measurable savings in bytes read from disk on the second and third query. This lesson builds, with the same evidence discipline, the opposite case: a cached DataFrame used only once. The memory cost is exactly as real as in lesson 5 — you're going to measure it with the same mechanism — but this time there's no second or third query taking advantage of it. The result is occupied memory, with no repeated work avoided, because there's no work that repeats.

Connection to the module. Lessons 5 and 6 showed .cache()/.persist()'s benefit and cost as separate pieces. This lesson brings both together: the same memory cost from lesson 6, but without lesson 5's benefit — the case this module's introduction promised from the start.

An analogy: the meal nobody eats again

Go back to the refrigerator. Imagine you prepare a special single-portion dish, for a visitor arriving just once. Storing it in the refrigerator costs exactly the same as storing any other meal — real space, no longer available for something else — but if that visitor never comes back, that space stayed occupied by something nobody else is going to eat. There's nothing wrong with cooking the dish or storing it — the mistake would be continuing to store single-portion dishes "just in case," until the refrigerator is full of food nobody touches, with no room for what actually does get reused.

Worked example: a one-off report, cached, and the cost it leaves behind

# single_use_cache.py
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from pyspark.sql.types import (
    StructType, StructField, StringType, IntegerType, DoubleType, TimestampType,
)

spark = SparkSession.builder.appName("kiosko-spark").master("local[*]").getOrCreate()

scale_schema = StructType([
    StructField("order_id", StringType(), False),
    StructField("franchise_id", IntegerType(), False),
    StructField("store_id", StringType(), False),
    StructField("product_id", StringType(), False),
    StructField("quantity", IntegerType(), False),
    StructField("unit_price", DoubleType(), False),
    StructField("order_ts", TimestampType(), False),
])
orders_at_scale_df = spark.read.csv("kiosko_orders_at_scale.csv", schema=scale_schema, header=True, enforceSchema=False)
fact = orders_at_scale_df.withColumn("revenue", F.col("quantity") * F.col("unit_price"))

# A one-off report: S02's revenue on a single day. Nobody else touches this DataFrame
# again for the rest of the script -- exactly the scenario this lesson tests.
s02_single_day = fact.filter((F.col("store_id") == "S02") & (F.col("order_ts") < "2026-08-04"))
s02_single_day.createOrReplaceTempView("s02_single_day")
s02_single_day.cache()

print("isCached BEFORE any action:", spark.catalog.isCached("s02_single_day"))

total = s02_single_day.agg(F.round(F.sum("revenue"), 2).alias("total")).collect()[0][0]
print(f"\ntotal S02, first day = {total}")
print("isCached AFTER the single query:", spark.catalog.isCached("s02_single_day"))

infos = spark.sparkContext._jsc.sc().getRDDStorageInfo()
print(f"\nCached RDDs: {len(infos)}")
for i in range(len(infos)):
    info = infos[i]
    print(f"    numCachedPartitions={info.numCachedPartitions()} memSize={info.memSize():,} diskSize={info.diskSize():,}")

print("\n-- this DataFrame never gets queried again for the rest of the script --")
spark.stop()

What to expect. Running python3 single_use_cache.py, the output is exactly this (executed in this run):

isCached BEFORE any action: True

total S02, first day = 975000.0
isCached AFTER the single query: True

Cached RDDs: 1
    numCachedPartitions=12 memSize=20,589,232 diskSize=0

-- this DataFrame never gets queried again for the rest of the script --

Check the result by hand before continuing: in KIOSKO_WEEK, S02 has two orders on 2026-08-03 — ORD-1003 (P003, revenue = 2 × 0.75 = 1.5) and ORD-1006 (P002, revenue = 2 × 1.20 = 2.4) — a total of 3.9 per franchise. Multiplied by the 250,000 synthetic franchises: 3.9 × 250,000 = 975,000.0, exactly the number the output shows. The real cost is in the last line before the comment: 20,589,232 bytes — nearly 20 MB — ended up stored in memory, marked as cached (isCached() == True), but the script ends with no second query ever touching them again.

Worked example, part 2: the same report, without .cache(), for a direct contrast

# single_use_no_cache.py -- identical, without the .cache() line
s02_single_day = fact.filter((F.col("store_id") == "S02") & (F.col("order_ts") < "2026-08-04"))
# no .cache() this time

total = s02_single_day.agg(F.round(F.sum("revenue"), 2).alias("total")).collect()[0][0]
print("total S02, first day (WITHOUT cache) =", total)

infos = spark.sparkContext._jsc.sc().getRDDStorageInfo()
print(f"Cached RDDs after the query: {len(infos)}")

What to expect (executed in this run):

total S02, first day (WITHOUT cache) = 975000.0
Cached RDDs after the query: 0

The result — 975000.0 — is identical with or without caching, because caching never changes what a query computes, only whether it stores the intermediate result for later. Without .cache(), after this single query, getRDDStorageInfo() reports zero cached RDDs: no memory block is occupied by this DataFrame, because it was never asked to be stored. The comparison is complete: with .cache(), you end up with 20,589,232 bytes occupied and no benefit; without .cache(), you end up with zero bytes occupied, and the same correct result.

Diagram: the same memory cost, with and without a benefit

flowchart TD
    subgraph L5["Lesson 5 -- fact reused across 3 queries"]
        A1["cache() + Q1: pays the cost\nof materializing"] --> A2["Q2: reuses -- saves ~602 MB of reads"]
        A2 --> A3["Q3: reuses -- saves ~602 MB of reads"]
        A3 --> A4["Net benefit: 2 complete reads avoided"]
    end

    subgraph L7["This lesson -- s02_single_day used once"]
        B1["cache() + single Q1: pays\nthe same kind of cost\n(20.6 MB retained)"] --> B2["(nobody else queries this DataFrame)"]
        B2 --> B3["Net benefit: ZERO --\nthe cost was paid for nothing in return"]
    end

    style A4 fill:#9c6,stroke:#333
    style B3 fill:#f96,stroke:#333

Going deeper: the cost isn't just the memory left occupied

It's worth naming two distinct costs, not just one. The first is the one you already measured: 20,589,232 bytes retained in memory, indefinitely, until something releases them — an explicit .unpersist(), or Spark evicting them under memory pressure. The second is less visible, but real: the extra work of writing that data to the cache area in the first place. When .cache() materializes a DataFrame for the first time, it doesn't just compute the result — it also serializes it (or prepares it deserialized, depending on the level) and writes it to Spark's storage area, a step a query with no caching never needs to take. In lesson 5, that extra cost gets amortized: it's paid once, and recovered many times over on the second and third queries. In this lesson, that same extra cost gets paid once, and is never recovered at all — it's pure overhead, with no future savings offsetting it.

Spark's official documentation is explicit about caching's memory management, and it's worth reading with this example in mind:

"Spark automatically monitors cache usage on each node and drops out old data partitions in a least-recently-used (LRU) fashion. If you would like to manually remove an RDD instead of waiting for it to fall out of the cache, use the RDD.unpersist() method."

This means Spark is eventually going to free s02_single_day's memory if it needs the space for something else, with nothing required from you — but "eventually" isn't free: meanwhile, that 20 MB competes for space with any other DataFrame that is actively being reused, and eviction by LRU is Spark's decision, not yours. The right discipline — the one exercise 1 of this lesson shows — is to explicitly release what you know you're not going to reuse, instead of letting the system discover it under memory pressure.

Common mistakes

Justifying a one-off DataFrame's cache with "it's small, it doesn't matter." What happens: someone caches a DataFrame of a few megabytes, reasoning that such a small cache can't cause any real problem, even if it's never reused. Why it happens: compared to fact_orders_at_scale's complete hundreds of megabytes, 20 MB seems insignificant. How to spot it: if your pipeline has several intermediate DataFrames, each cached "because it's small" and used only once, add up those costs — a pipeline with ten one-off reports of 20 MB each, all cached unnecessarily, retains 200 MB of memory with no benefit at all, the same class of cost as a single large DataFrame with no reuse. How to fix it: this lesson's criterion doesn't depend on absolute size — it depends on whether there's real reuse. A 20 MB DataFrame used once still doesn't justify caching, no matter how small it is compared to other data in the same pipeline.

Forgetting to release a one-off report's cache in a notebook or long-running session. What happens: in an interactive session — a notebook, for example — someone caches several DataFrames throughout an exploratory investigation, each one to answer a specific question, and never calls .unpersist() on any of them. Why it happens: in a script that ends and calls spark.stop(), all the memory gets released automatically at the end; in a session that stays alive for hours, that "end" never comes, and retained memory piles up. How to spot it: if your Spark session has been active for a long time and you don't remember how many different DataFrames you cached throughout it, check spark.sparkContext._jsc.sc().getRDDStorageInfo() — as this lesson did — to see the complete list of what's still retained. How to fix it: in long-running sessions, adopt the discipline of .unpersist() as soon as you're done with a one-off DataFrame, instead of trusting Spark to evict it by LRU right when it's needed — this lesson's "going deeper" section's official quote confirms that automatic eviction exists, but it depends on memory pressure triggering it, not on your intent.

Measuring caching's "benefit" by counting how many times .cache() shows up in the code, instead of how many times the result gets reused. What happens: someone reviews a pipeline and concludes it's well optimized because it has many lines with .cache(), without checking, for each one, how many later queries reuse that specific result. Why it happens: .cache() visually feels like a signal that "performance was considered" — more lines with that word superficially look like more optimization. How to spot it: for every .cache() in a pipeline, ask yourself: how many distinct actions, after this line, touch this same object? If the answer is one, that specific line isn't optimizing anything — it's paying this lesson's cost with none of lesson 5's benefit. How to fix it: audit every .cache() in a real pipeline by counting real reuse, not the method's presence. This guide demonstrated both cases with evidence — three queries that do benefit (lesson 5), one that doesn't (this lesson) — precisely so the criterion is counting reuse, not counting lines of code.

Exercises

Exercise 1 — Explicitly release s02_single_day as soon as you're done using it, and confirm the release. On the worked example's cached DataFrame, call .unpersist(blocking=True) right after obtaining total, and confirm with getRDDStorageInfo() that the memory got released.

total = s02_single_day.agg(F.round(F.sum("revenue"), 2).alias("total")).collect()[0][0]
print(f"total = {total}")

s02_single_day.unpersist(blocking=True)

infos = spark.sparkContext._jsc.sc().getRDDStorageInfo()
print(f"Cached RDDs after unpersist(): {len(infos)}")

total = 975000.0
Cached RDDs after unpersist(): 0

Exercise 2 — Simulate three different one-off reports (three stores, three different days), each cached and used only once, and add up the total cost. Repeat this lesson's pattern for S01 and S03 besides S02, each with .cache(), each queried only once, and sum the three memSize values to see this practice's accumulated cost.

totals = {}
mem_sizes = []
for store in ["S01", "S02", "S03"]:
    single_day = fact.filter((F.col("store_id") == store) & (F.col("order_ts") < "2026-08-04"))
    single_day.cache()
    totals[store] = single_day.agg(F.round(F.sum("revenue"), 2).alias("total")).collect()[0][0]

infos = spark.sparkContext._jsc.sc().getRDDStorageInfo()
for i in range(len(infos)):
    mem_sizes.append(infos[i].memSize())

print("totals:", totals)
print("cached RDDs:", len(infos))
print("total memory retained:", sum(mem_sizes), "bytes")

totals: {'S01': ..., 'S02': 975000.0, 'S03': ...}
cached RDDs: 3
total memory retained: ... bytes (the sum of all three, none reused)

Exercise 3 — Explain, without code, why the result (975000.0) is identical with and without .cache(), but memory's state after the query isn't. In 2-3 sentences, explain why caching never changes a query's result, only its later cost.

Summary and next step

This lesson completed this module's caching criterion with the opposite case from lesson 5: s02_single_day, cached and used only once, retained 20,589,232 bytes in memory — the same kind of cost, measured with the same mechanism (getRDDStorageInfo()) — with no second or third query taking advantage of it. The direct contrast, without caching, confirmed the result is identical (975000.0) and memory drops to zero after the query. This module's complete criterion stands, with evidence from both sides: caching helps when there's real reuse (lesson 5), and only costs memory when there isn't (this lesson) — the same decision, applied to the same data, with opposite results based on a single question: how many more times is this going to get used?

Before moving on you should be able to: explain why a cache's memory cost is independent of whether it gets reused or not; count, for any .cache() in a real pipeline, how many later queries justify it; and decide, with that count, whether to cache or not before writing the line of code.

Lesson 8 integrates this module's previous seven lessons — Catalyst's four phases, .explain()'s five modes, Adaptive Query Execution, and the complete caching criterion — into a single pipeline over fact_orders_at_scale, verified end to end against the known total.

Resources

• Apache Spark — RDD Programming Guide, "Removing Data" section (automatic LRU eviction behavior, and the explicit recommendation to use unpersist() to free memory manually — the exact quote used in this lesson's "going deeper" section). spark.apache.org/docs/latest/rdd-programming-guide.html#removing-data.
• PySpark — DataFrame.unpersist (official API reference, including the blocking parameter). spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.unpersist.html.
• This guide's DESIGN doc (spark-and-distributed-processing-guide/DISENO.md) — this lesson's exact specification: a deliberate case of caching that doesn't help, to show the memory cost with no benefit.