Naming Structured Streaming, without building it

Description

This is, deliberately, the shortest lesson in this entire guide, and the only one that runs not a single line of code. Its only job is to name something, precisely and with the correct official quote, without building anything on top of it: Structured Streaming, the extension of the same DataFrame API you mastered in modules 1 through 7 toward data arriving continuously, instead of in batches. Nothing you did in this guide — reading with an explicit schema, .select(), .filter(), .groupBy(), .join(), window functions, even partitioned Parquet — stops applying once the data stops being a fixed file. What changes is the arrival rhythm, and this lesson only names that extension, without writing a real streaming pipeline.

Connection to the module. This lesson adds no new mechanism to what you already know about Spark — no new .explain() to read, no plan to interpret. It's, deliberately, a map toward a sibling guide, not a construction. streaming-with-kafka-and-flink-guide is the complete guide that does build a real streaming pipeline, with a Kafka source, watermarks, and continuous triggers — none of that shows up here.

An analogy: the same inventory, two arrival rhythms

This entire module worked like someone receiving Kiosko's weekly inventory in a single truck, once a week: the truck arrives complete, you count and classify everything it brought, and you're done — the same pattern you've used since module 1, with the seven CSV files from a fixed week. Structured Streaming is the same warehouse, but with a conveyor belt that never stops: instead of waiting for the complete truck before starting to work, every box gets processed as soon as it arrives, with exactly the same counting and classification rules you already wrote for the weekly truck. The business criteria don't change one bit — they're still margin_category's same rules, the same groupBy("store_id") — what changes is that the work no longer has a natural end: the conveyor belt, in principle, never stops moving.

Comparison, not run: the same verbs, two ways to arrive

Instead of a worked example with code — the one moment in this guide where that doesn't apply — this lesson compares, in a table, the exact vocabulary you already used in modules 1 through 7 (left) against Structured Streaming's equivalent vocabulary (right), as documented by Spark's official guide. Nothing in the right column runs in this lesson — it's a reference, not an exercise.

Concept (already used in this guide)	Batch API — what you ran, modules 1-7	Streaming API — named here, not built
Read entry point	spark.read.csv(...) (module 1)	spark.readStream.csv(...)
What the read returns	DataFrame	DataFrame — the same class
Transformations	.select(), .filter(), .groupBy(), .join(), Window (modules 2 through 6)	The same ones, without changing a single line of that logic
Triggering the work	.count(), .collect(), .show() — one action, once (module 2)	.writeStream.start() — a continuous query, not a one-time action
Writing the result	.write.parquet(...) (modules 3 and 7)	.writeStream.format(...).start()
The plan Catalyst runs	The same optimizer (module 6)	The same optimizer — Catalyst doesn't distinguish between the two APIs

The row most worth underlining is the last one: Structured Streaming isn't a different engine running underneath — it's the same Spark SQL engine, the same Catalyst you learned to read in module 6, applied to a DataFrame that, instead of having a known end, keeps updating with new data.

Diagram: the same trunk, two branches

flowchart TD
    A["DataFrame API\n(modules 1-7 of this guide:\nselect, filter, groupBy, join, Window,\nCatalyst, explain(), pandas_udf)"] --> B["spark.read...\nBatch: a DataFrame with a known end\n(EVERYTHING you ran in this guide)"]
    A --> C["spark.readStream...\nStreaming: a DataFrame that updates\ncontinuously (named here, NOT built)"]
    C -.->|"real source (Kafka),\nwatermarks, triggers"| D(("streaming-with-kafka-and-flink-guide"))

Going deeper: the official quote, and what this lesson does NOT cover

Spark's official documentation defines Structured Streaming like this: "Structured Streaming is a scalable and fault-tolerant stream processing engine built on the Spark SQL engine". That same documentation adds the sentence fully backing this lesson: "You can express your streaming computation the same way you would express a batch computation on static data" — and, even more specifically: "You can use the Dataset/DataFrame API in Scala, Java, Python or R to express streaming aggregations, event-time windows, stream-to-batch joins, etc. The computation is executed on the same optimized Spark SQL engine".

It's worth being explicit about what this lesson does not cover, so as not to leave a false sense of completeness. It builds no real streaming source — Kafka, sockets, or any messaging system — no watermark (the mechanism for handling data that arrives late), no trigger (how often a stream's micro-batches get processed), no fault-tolerance handling with checkpointing. All four are real, necessary pieces of any production streaming system, and all four are, explicitly, the complete content of streaming-with-kafka-and-flink-guide — the sibling guide that does build them, with a real Kafka cluster. It's also worth noting Spark has, besides Structured Streaming, an older API called Spark Streaming, based on DStreams — built on RDDs, not on DataFrame — documented separately in the official guide. This guide, consistent with module 2's decision to leave RDDs behind as the main interface, names only Structured Streaming: it's the modern API, built on exactly the same DataFrame API you mastered across seven complete modules.

Common mistakes

Finishing this lesson thinking you already know how to build a production streaming pipeline. What happens: someone reads this lesson's comparison table, sees the verbs are nearly identical to the ones they already master, and concludes they "already know streaming" in Spark. Why it happens: the vocabulary's similarity is real and deliberate — it's exactly this lesson's point — but it can suggest a deeper familiarity than actually exists. How to spot it: if you can't explain what a watermark is or why one is needed to handle out-of-order data, you don't yet have complete streaming knowledge — you only have the vocabulary shared with batch. How to fix it: this lesson honestly gives you ten percent of the picture — "the same API, applied to continuous data" — the remaining ninety percent — real sources, fault tolerance, delivery semantics, handling late data — is streaming-with-kafka-and-flink-guide's complete content, not something you can infer from this one table.

Confusing Structured Streaming with Spark's older Spark Streaming (DStreams) API. What happens: someone searches for information about streaming in Spark, finds documentation or tutorials about the DStreams API — older, based on RDDs — and mixes it up with what this lesson names. Why it happens: both share the word "Streaming" in the name, and Spark's official documentation still keeps separate pages for the two. How to spot it: if the code you find uses StreamingContext or DStream instead of SparkSession.readStream, you're looking at the old API, based on RDDs — the same abstraction this guide's module 2 used just once, for historical contrast, and never built on again. How to fix it: Structured Streaming, the only one this lesson names, is built on the DataFrame API — this entire guide's same DataFrame — not on RDDs; if you ever need real streaming in a project of your own, Structured Streaming is the modern API to use, not DStreams.

Looking for .writeStream.start()'s result as if it were a normal action, with an immediate return value. What happens: someone, reasoning by analogy with .write.parquet(...) (an action that finishes and returns control), expects .writeStream.format(...).start() to also finish quickly and return a final result. Why it happens: the rest of this guide, since module 2, taught that an action (.count(), .collect(), .write...) triggers the work and finishes. How to spot it: this lesson's table already flags it — .writeStream.start() starts a continuous query, which keeps running indefinitely until something explicitly stops it, not a one-time action. How to fix it: there's nothing to fix in this lesson — no code runs — but it's worth keeping this distinction in mind for when you reach streaming-with-kafka-and-flink-guide: a streaming query lives continuously in the SparkSession, with a lifecycle completely different from a batch action's.

Exercises

Exercise 1 — Explain, without code, why this lesson's table's "The plan Catalyst runs" row is the most important of the six. In 2-3 sentences, using what you already know about Catalyst from module 6, explain why the fact that Structured Streaming uses the same optimizer is the underlying reason the rest of the table makes sense.

See solution

If Structured Streaming used a different optimizer, the table's vocabulary similarity (.select(), .filter(), .groupBy()) would just be a superficial naming coincidence, with no guarantee the logic would behave the same between batch and streaming. The fact that both share the same Spark SQL engine — the same Catalyst you learned to read in module 6, with the same optimization phases — is what guarantees an expression like .groupBy("store_id").agg(F.sum("revenue")), written once, produces consistent results regardless of whether the DataFrame comes from spark.read or from spark.readStream. It's the structural reason, not just a stylistic one, why the rest of the table — the shared verbs — makes sense.

Exercise 2 — Explain, without code, the difference between a watermark and a trigger, based only on their names and this lesson's context (without looking up the complete documentation). In 2-3 sentences, speculate with reasonable judgment — it doesn't need to be perfect — what problem each one solves, given both got named in this lesson's "going deeper" section as pieces streaming-with-kafka-and-flink-guide is going to build.

See solution

A reasonable hypothesis, based only on the names: a trigger probably controls when Spark processes the next batch of new data in a continuous stream — how often, or under what condition, the next processing cycle fires. A watermark, in the context of data arriving late or out of order, probably defines up to what point in time Spark still waits for late data before considering a time window "closed" and stops waiting for more. This hypothesis doesn't need to be exact — the complete, verified answer is streaming-with-kafka-and-flink-guide's content — the exercise is practicing reasoning about a new concept with the available context, before looking up the complete answer in the right source.

Exercise 3 — Explain, without code, why this guide chose to name Structured Streaming instead of simply omitting it entirely. In 2-3 sentences, and considering this guide's explicit boundary with streaming-with-kafka-and-flink-guide, explain what value a "just name it" lesson has within a guide that, across the rest of its seven modules, always runs real code.

See solution

Omitting Structured Streaming entirely would leave an open question unanswered: after mastering the DataFrame API in depth across seven modules, it would be reasonable to wonder whether that API serves any purpose beyond static files, and without this lesson, that question would stay unresolved within this guide. Naming it, without building it, gives whoever finishes this guide a correct map — they know it exists, know it shares the same API and the same optimizer, and know exactly which sibling guide to continue in if they need real streaming — without inflating this guide's scope toward a topic that deserves its own complete treatment (Kafka, watermarks, fault tolerance). It's the same boundary discipline this guide holds with Iceberg/Delta Lake, dbt, and Airflow: naming where the scope ends, without pretending the topic doesn't exist nor building it halfway.

Summary and next step

This lesson named, without building anything, Structured Streaming: the same DataFrame API you mastered in modules 1 through 7 — schema-based reading, .select(), .filter(), .groupBy(), .join(), Window, Parquet, the same Catalyst optimizer — applied to data arriving continuously instead of in batches. You confirmed, with the official quote, that the same Spark SQL engine processes both cases, and you made the boundary clear: real sources, watermarks, triggers, and fault tolerance are streaming-with-kafka-and-flink-guide's complete content, not this lesson's.

Before moving on you should be able to: name from memory the equivalent verbs between batch and streaming (read/readStream, action/writeStream.start()); explain why Structured Streaming is built on DataFrame, not on RDDs; and explain, in your own words, what this lesson did NOT cover.

Lesson 8 — the project closing out this module — returns to fully executed ground: it integrates partitioned Parquet, partition pruning, predicate pushdown, and pandas_udf, all over fact_orders_at_scale's ten million rows, in a single pipeline verified end to end.

Resources

• Apache Spark — Structured Streaming Overview (the complete official quote: "Structured Streaming is a scalable and fault-tolerant stream processing engine built on the Spark SQL engine", and confirmation it uses the same DataFrame API as batch computation). spark.apache.org/docs/latest/streaming/index.html.
• This guide's DESIGN doc (spark-and-distributed-processing-guide/DISENO.md) — the explicit boundary: "Real-time structured streaming... here Structured Streaming (M7) gets named as the API that reuses the same engine and the same DataFrame API for continuous data — one paragraph, no streaming code, no Kafka cluster".
• streaming-with-kafka-and-flink-guide — this ecosystem's sibling guide that builds real streaming: a Kafka source, watermarks, triggers, fault tolerance. src/guides/streaming-with-kafka-and-flink-guide/