Module 6: Merge Into And Native Upserts

Setting up Spark with the Iceberg runtime

Description

This lesson leaves theory behind for the rest of the module: it really installs the piece a plain Spark is missing to talk to Iceberg tables — the Iceberg runtime for Spark — and confirms, with real evidence and not a promise, exactly how far that installation gets in this guide's environment. You're going to see the correct jar download from Maven Central, the SparkSession start up with the exact version you expect, and you're also going to see a real, documented error that exists in the Iceberg ecosystem itself right now — and why that error, far from being this lesson's failure, is exactly the kind of evidence this guide promises to show instead of hide.

Connection to the module. Lesson 1 explicitly stated this is the only module in the whole guide that needs the JVM. This lesson delivers on that promise: it installs Spark with the Iceberg runtime, reusing the PySpark 4.2.0 + Java 17 that spark-and-distributed-processing-guide already left configured. Lessons 4 and 5 build on exactly what this lesson leaves working — and on exactly what this lesson discovers does not work yet.

An analogy: the translator who speaks your language, not the bank's

Going back to the module's analogy: the bank counter (MERGE INTO) exists, but it needs someone staffing it with the right language. Spark, on its own, doesn't know how to read an Iceberg table — it understands Parquet as a loose file, but it doesn't understand the metadata → manifest list → manifest files → data files chain this guide's module 2 already mapped. The Iceberg runtime for Spark is the translator hired for that specific window: a code package, published by the Apache Iceberg project itself, that teaches Spark to read and write the full table format, not just the loose Parquet files inside it.

Step 1 — Verify the inherited environment: Java 17 and PySpark

spark-and-distributed-processing-guide (module 1, lesson 4) already left Java 17 and PySpark installed and verified. Confirm it on your own machine before moving on:

java -version
echo $JAVA_HOME
python3 -c "import pyspark; print('pyspark', pyspark.__version__)"

What to expect (verified on this machine; JAVA_HOME's exact path is specific to this computer — macOS, installed with Homebrew — and it's going to look different on yours, the exact same warning spark-and-distributed-processing-guide gave at the time):

openjdk version "17.0.20" 2026-07-21
OpenJDK Runtime Environment Homebrew (build 17.0.20+0)
OpenJDK 64-Bit Server VM Homebrew (build 17.0.20+0, mixed mode, sharing)

/opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home

pyspark 4.2.0

If your JAVA_HOME comes back empty, or your Java version is lower than 17, go back to spark-and-distributed-processing-guide's module 1, lesson 4 — this lesson doesn't repeat that installation, it takes it as given.

Step 2 — Verify the runtime's exact coordinate on Maven Central

Before writing a single line of configuration, confirm the org.apache.iceberg:iceberg-spark-runtime-4.0_2.13:1.11.0 coordinate is real and current — this guide's DESIGN doc explicitly warns these artifacts rotate with every Iceberg release, so it isn't taken as given:

curl -s https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-runtime-4.0_2.13/maven-metadata.xml

What to expect (verified against real Maven Central while writing this lesson):

<metadata>
  <groupId>org.apache.iceberg</groupId>
  <artifactId>iceberg-spark-runtime-4.0_2.13</artifactId>
  <versioning>
    <latest>1.11.0</latest>
    <release>1.11.0</release>
    <versions>
      <version>1.10.0</version>
      <version>1.10.1</version>
      <version>1.10.2</version>
      <version>1.11.0</version>
    </versions>
    <lastUpdated>20260519045253</lastUpdated>
  </versioning>
</metadata>

Confirmed: 1.11.0 is, right now, iceberg-spark-runtime-4.0_2.13's latest and release version — the same coordinate this guide's DESIGN doc anticipated, now verified against the real repository, not assumed. 4.0_2.13 means: compiled for Spark 4.0, with Scala 2.13 — the same Scala version pyspark==4.2.0 ships with (you can confirm it by listing your PySpark installation's .jars: spark-core_2.13-4.2.0.jar, spark-sql_2.13-4.2.0.jar). Notice the artifact's Spark number — 4.0 — compared against your installed PySpark's number — 4.2.0: they aren't the same number. Hold onto that observation; the rest of this lesson explains why it matters.

Step 3 — Build the `SparkSession` with the Iceberg runtime

# spark_iceberg_session.py
import os

os.environ["JAVA_HOME"] = "/opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home"  # your path may vary

from pyspark.sql import SparkSession

warehouse_path = os.path.abspath("kiosko_spark_warehouse")

spark = (
    SparkSession.builder
    .appName("kiosko-iceberg-merge")
    .config("spark.jars.packages", "org.apache.iceberg:iceberg-spark-runtime-4.0_2.13:1.11.0")
    .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
    .config("spark.sql.catalog.local", "org.apache.iceberg.spark.SparkCatalog")
    .config("spark.sql.catalog.local.type", "hadoop")
    .config("spark.sql.catalog.local.warehouse", f"file://{warehouse_path}")
    .getOrCreate()
)

print("SPARK VERSION:", spark.version)

Every .config(...) line comes, literally, from Iceberg's official Spark documentation — none of these are values invented for this guide: spark.jars.packages is Spark's standard mechanism for pulling a Maven Central dependency at startup, with no need to install anything by hand beforehand; spark.sql.extensions enables the additional SQL syntax Iceberg adds to Spark (including MERGE INTO on Iceberg tables); spark.sql.catalog.local with type=hadoop registers a catalog named local — the same name the official Iceberg "Getting Started" documentation uses, verbatim — backed only by the filesystem, with no external database.

Stop on the name local. It's the same literal convention from the official Iceberg documentation, and this guide deliberately respects it — but it's a completely distinct catalog from the kiosko catalog PyIceberg used in modules 1 through 5: a different registration mechanism (hadoop, file-based, versus sql/SQLite), a different warehouse directory (kiosko_spark_warehouse/ versus kiosko_warehouse/). Any local.kiosko.dim_product table you create in this module is a new table, independent of kiosko.dim_product — never the same table seen from two engines.

Step 4 — Run the session: what does work

python3 spark_iceberg_session.py

What to expect (verified by running the real script, with network access to Maven Central):

:: loading settings :: url = jar:file:/.../pyspark/jars/ivy-2.5.3.jar!/org/apache/ivy/core/settings/ivysettings.xml
Ivy Default Cache set to: /Users/.../.ivy2.5.2/cache
The jars for the packages stored in: /Users/.../.ivy2.5.2/jars
org.apache.iceberg#iceberg-spark-runtime-4.0_2.13 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-...;1.0
  confs: [default]
  found org.apache.iceberg#iceberg-spark-runtime-4.0_2.13;1.11.0 in central
downloading https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-runtime-4.0_2.13/1.11.0/iceberg-spark-runtime-4.0_2.13-1.11.0.jar ...
  [SUCCESSFUL ] org.apache.iceberg#iceberg-spark-runtime-4.0_2.13;1.11.0!iceberg-spark-runtime-4.0_2.13.jar (1328ms)
:: resolution report :: resolve 394ms :: artifacts dl 1331ms
  1 artifacts copied, 0 already retrieved (46803kB/34ms)
Setting default log level to "WARN".
SPARK VERSION: 4.2.0

This is real, not representative: the iceberg-spark-runtime-4.0_2.13-1.11.0 .jar — 46.8 MB — really got downloaded from Maven Central, spark.jars.packages worked exactly as Iceberg documents, and the SparkSession started up with the exact PySpark version this guide inherits: 4.2.0. Up to here, everything works.

Step 5 — What doesn't work, yet, with real evidence

The natural next step would be to create the kiosko namespace inside the local catalog and start working. Run that line, on the same session:

spark.sql("CREATE NAMESPACE IF NOT EXISTS local.kiosko")

What to expect (the real, verified error — not invented — abbreviated version of the full traceback):

Traceback (most recent call last):
  ...
py4j.protocol.Py4JJavaError: An error occurred while calling o40.sql.
: java.lang.IncompatibleClassChangeError: class org.apache.iceberg.spark.source.SparkView
  can not implement org.apache.spark.sql.connector.catalog.View, because it is not an
  interface (org.apache.spark.sql.connector.catalog.View is in unnamed module of loader 'app')
		at java.base/java.lang.ClassLoader.defineClass1(Native Method)
		...
		at org.apache.spark.sql.connector.catalog.Catalogs$.load(Catalogs.scala:65)
		at org.apache.spark.sql.connector.catalog.DefaultCatalogManager.$anonfun$catalog$1(CatalogManager.scala:134)
		...

And it isn't an isolated case for CREATE NAMESPACE: the same error shows up with any operation on the local catalog, including one as simple as SHOW NAMESPACES IN local — the error happens while loading the catalog's own class, before the specific operation even matters.

Why this happens, and why it gets documented instead of hidden

This isn't a configuration error in this lesson — Java 17 is properly installed, JAVA_HOME points at the right place, the Maven coordinate is the exact one this guide's DESIGN doc asked to be verified. It's a real, documented binary incompatibility between iceberg-spark-runtime-4.0 and Spark 4.1 onward. Apache Iceberg's own GitHub repository has an open issue about exactly this family of problem: apache/iceberg#15238, "Spark 4.1 incompatible: Create View," opened on February 5, 2026 and still open at the time this lesson was written. The root cause that issue describes: Spark 4.1 added a third parameter to the internal ResolvedIdentifier class (output: Seq[Attribute] = Nil), a change that breaks binary compatibility with code Iceberg compiled against Spark 4.0's earlier signature — the same family of breakage that produces the IncompatibleClassChangeError you see above, on the SparkView class.

And there's a second piece of evidence, verifiable with this same lesson's Step 2: at the time this guide was written, iceberg-spark-runtime doesn't publish any 4.1 or 4.2 variant on Maven Central — only 4.0_2.13 exists for the Spark 4.x family. pyspark==4.2.0, the one this guide inherits from spark-and-distributed-processing-guide, is already two minor versions ahead of the most recent Iceberg runtime available. This isn't a bug in this guide or on your machine — it's the real state of the Iceberg-for-Spark artifact ecosystem right now, exactly the warning this guide's DESIGN doc anticipated: "these artifacts rotate with every release."

The practical consequence for the rest of this module, stated that explicitly: lessons 4 and 5 show the real MERGE INTO, with the exact syntax you'd run if your Spark and Iceberg version combination were compatible, but its output gets marked as "What to expect (representative)" — verified against Iceberg's official documentation, not against a real run in this environment. Lesson 6, on the other hand, does run for real: PyIceberg's table.upsert() doesn't depend on Spark or on this incompatibility, so its output is literal, executed, with no caveats.

Diagram: how far this lesson got

flowchart TD
    A["Java 17 + JAVA_HOME\nverified, works"] --> B["pyspark 4.2.0\nverified, works"]
    B --> C["spark.jars.packages downloads\niceberg-spark-runtime-4.0_2.13:1.11.0\nverified, works"]
    C --> D["SparkSession starts up\nspark.version == 4.2.0\nverified, works"]
    D --> E["spark.sql('CREATE NAMESPACE ...')\nIncompatibleClassChangeError\nREAL, documented in iceberg#15238"]
    E -.->|"lessons 4 and 5"| F["MERGE INTO: real syntax,\n'What to expect (representative)' output"]
    E -.->|"lesson 6, no Spark"| G["table.upsert(): really\nexecuted, no caveats"]

Common mistakes

Assuming an iceberg-spark-runtime-X_Y coordinate works with any Spark version that starts with the same major number. What happens: someone sees iceberg-spark-runtime-4.0_2.13 and pyspark==4.2.0, and assumes "4.0" and "4.2" are compatible because both are "Spark 4." Why it happens: in many Python libraries, a minor version change (4.0 to 4.2) is usually backward compatible with no friction. How to spot it: if your MERGE INTO or any Iceberg catalog operation fails with IncompatibleClassChangeError or NoSuchMethodError mentioning classes from org.apache.spark.sql.connector.catalog, suspect a version mismatch between the Iceberg runtime and your exact Spark first, before reviewing your own SQL code. How to fix it: always verify, like this lesson's Step 2 did, which iceberg-spark-runtime variants exist on Maven Central for your exact Spark version — if none matches exactly, as is the case for Spark 4.2.0 right now, the real option is using a Spark version that does have a compatible runtime (4.0.x), or waiting for Iceberg to publish the matching variant.

Confusing this error with a Java or JAVA_HOME problem. What happens: someone sees a long traceback, with Java classes on every line, and assumes the problem is the Java installation — they go back and check JAVA_HOME, reinstall the JDK, with no result. Why it happens: any error showing java.lang.* on the first line instinctively feels like a problem with the Java installation itself. How to spot it: look at this lesson's Step 4 — the SparkSession started up with no problem at all, with spark.version confirmed, before the error showed up. If your session starts fine and the error only appears when running SQL against the Iceberg catalog, Java is installed correctly — the problem is a binary incompatibility between library versions, not your JDK. How to fix it: don't reinstall Java — confirm, like this lesson did, that the error happens specifically while loading org.apache.iceberg.spark.* classes, and treat it for what it is: a version mismatch between Iceberg and Spark, documented in apache/iceberg#15238.

Assuming a real error invalidates the rest of the module. What happens: someone, on seeing MERGE INTO doesn't really run in this environment, assumes lessons 4 and 5 have no value, or that they should be skipped. Why it happens: it's easy to think a technical lesson is only worthwhile if the code ran with no errors. How to spot it: if your reaction to this lesson is "so this part of the guide is useless," reread the previous section — the exact MERGE INTO syntax lessons 4 and 5 teach is real, verified against the official documentation, and it's exactly what you'd run in an environment with compatible versions (for example, Spark 4.0.x with this same runtime). How to fix it: treat this lesson's error for what it is — real evidence that an active ecosystem's artifacts rotate and sometimes fall out of alignment, something you're going to run into in any real data stack. The syntax you learn in lessons 4 and 5 stays correct and applicable the day your version combination does become compatible.

Exercises

Exercise 1 — Reproduce the Maven Central verification yourself. Run this lesson's Step 2 curl command. Confirm 1.11.0 is still the latest version, or identify if a newer one has already come out.

See solution

If you ran this exercise after this guide was written, you might see a version newer than 1.11.0 in the <latest> field — Iceberg publishes releases regularly. That doesn't invalidate this lesson: check whether that newer version already includes a 4.1_2.13 or 4.2_2.13 variant of iceberg-spark-runtime — if so, this lesson's incompatibility might be resolved for your version combination, and it would be worth repeating Step 4 with the updated coordinate.

Exercise 2 — Explain, without looking at the lesson, the difference between "the Spark session starts" and "the Iceberg catalog works." In 2-3 sentences, explain why this lesson's Step 4 succeeded (spark.version confirmed) while Step 5 failed, even though both run on the same SparkSession.

See solution

Starting a SparkSession with spark.jars.packages only needs Spark to download the .jar and add it to its classpath — that step doesn't run any Iceberg-specific code yet, so it can succeed even if the .jar's contents are incompatible. The error only shows up when Spark tries to load and use a specific class from the Iceberg runtime — like SparkCatalog, while resolving local.kiosko — the moment at which the JVM verifies, at runtime, that the classes are binarily compatible with each other. IncompatibleClassChangeError is, precisely, the error the JVM throws when that verification fails — never before, because up until then nobody had asked the JVM to use that specific class.

Exercise 3 — Prediction. With this lesson's evidence, predict: if you installed pyspark==4.0.0 instead of pyspark==4.2.0 (an exact version that does match iceberg-spark-runtime-4.0_2.13), do you expect this lesson's Step 5 to work with no IncompatibleClassChangeError?

See solution

It's reasonable to expect so — the runtime's coordinate explicitly says 4.0, and issue apache/iceberg#15238 describes the break as something introduced specifically in Spark 4.1, not present in Spark 4.0. This guide doesn't verify that exact combination, because doing so would mean reinstalling a Spark version different from the one the rest of the ecosystem (spark-and-distributed-processing-guide) already left configured, breaking the environment reuse this guide's DESIGN doc explicitly asks for. It stays a testable hypothesis, not a fact confirmed in this lesson.

Summary and next step

In this lesson you really installed the environment this module needs: you confirmed Java 17 and PySpark 4.2.0 inherited from spark-and-distributed-processing-guide, verified the exact iceberg-spark-runtime-4.0_2.13:1.11.0 coordinate against real Maven Central, watched the .jar download and the SparkSession start up successfully — and also saw, with a real traceback, a documented binary incompatibility (apache/iceberg#15238) between that runtime and Spark 4.1 onward, which makes the next two lessons present their MERGE INTO as representative instead of executed.

Before moving on you should be able to: explain the difference between "the session starts" and "the catalog works"; name the incompatibility's exact root cause (the ResolvedIdentifier signature change in Spark 4.1); and explain why this lesson documents the error instead of omitting it.

Lesson 4 uses exactly this environment — incompatibility included, stated that explicitly — to teach MERGE INTO's general syntax on Iceberg, verified against the official documentation.

Resources

Apache Iceberg — official documentation, "Getting Started," "Using Iceberg in Spark" section — literal source of the local catalog convention and spark.jars.packages. iceberg.apache.org/docs/latest/getting-started. In English.
Apache Iceberg — official documentation, "Spark Configuration" — source of spark.sql.catalog.<name>.type=hadoop and spark.sql.extensions. iceberg.apache.org/docs/latest/spark-configuration. In English.
Maven Central — iceberg-spark-runtime-4.0_2.13, metadata verified in this lesson's Step 2. repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-runtime-4.0_2.13. In English.
GitHub — apache/iceberg#15238, "Spark 4.1 incompatible: Create View" — the real, open issue documenting this lesson's error's root cause. github.com/apache/iceberg/issues/15238. In English.
spark-and-distributed-processing-guide DESIGN doc — source of the PySpark 4.2.0 + Java 17 environment this lesson reuses. src/guides/spark-and-distributed-processing-guide/DISENO.md. In Spanish.
This guide's DESIGN doc — the explicit warning about Iceberg artifacts rotating, and the rule to verify them while writing each lesson. src/guides/lakehouse-and-iceberg-guide/DISENO.md. In Spanish.

← PreviousThree ways Kiosko already solved this Next →`MERGE INTO` in Spark SQL

Setting up Spark with the Iceberg runtime

Description

An analogy: the translator who speaks your language, not the bank's

Step 1 — Verify the inherited environment: Java 17 and PySpark

Step 2 — Verify the runtime's exact coordinate on Maven Central

Step 3 — Build the SparkSession with the Iceberg runtime

Step 4 — Run the session: what does work

Step 5 — What doesn't work, yet, with real evidence

Why this happens, and why it gets documented instead of hidden

Diagram: how far this lesson got

Common mistakes

Exercises

Summary and next step

Resources

Step 3 — Build the `SparkSession` with the Iceberg runtime