Catalogs beyond local: REST, Glue, Unity Catalog, Polaris

Description

Since module 2, this guide has used a catalog named kiosko, backed by a SQLite file and a warehouse on your own filesystem. It worked perfectly: it cost $0, needed no account, and did its one job — always pointing at each table's current metadata file — without failing once across six modules. This lesson doesn't replace that catalog. It names, one by one, the four catalogs that really get used when that same job has to hold up with many different engines, writing at once, from different machines — without implementing any of them, no cloud account, no credentials.

Connection to the module. Lesson 1 promised to name the production catalogs "by their guarantee, without implementing them." This lesson delivers on that: first it pins down exactly what guarantee any Iceberg catalog solves — the same one kiosko already gave you, without you noticing — and then it names the four that sustain it at production scale.

The guarantee you already had, without knowing it had a name

Go back to module 2, lesson 2: the kiosko catalog stores, for kiosko.fact_orders, one file's exact path — the current metadata.json. Every time you wrote something new to that table — every append(), overwrite(), upsert() from modules 1 through 6 — PyIceberg did two things, always in this order: it wrote a new, complete metadata file, without touching the previous one; and then, only if that write finished with no errors, it asked the catalog to update its pointer to point at the new file. That second operation — "update the pointer, but only if nobody else changed it while I was writing" — is a conditional atomic update (compare-and-swap: "compare the current value against what I expected, and only then replace it"). It's the minimum, non-negotiable guarantee of any Iceberg catalog, whether it's kiosko/SQLite or the most sophisticated of the four you're going to learn about in this lesson.

Why does "conditional" matter? Because if two writes finish almost at the same time, without that condition the catalog could accept both, and the second would silently overwrite the first one's work. With the condition, the second write discovers, at the moment it tries to update the pointer, that the value is no longer what it expected — someone else got there first — and its commit fails, forcing it to retry against the most recent state. SQLite, being a real transactional database, already gave you this guarantee across the previous six modules, even though you never actually tested it with two writers really competing — because, in this whole guide, the only writer was your own script.

An analogy: the same counter, now with a hundred windows

The bank counter from module 6's lesson 1 handled one transaction at a time, with one customer in front of it. A production catalog is that same counter, but with a hundred windows open at the same time, all updating the same bank account at the same instant — a Spark engine running a MERGE INTO, a Python pipeline doing upsert(), an analyst querying from a notebook, all reading or writing the same Iceberg table, maybe from different continents. The bank needs something the single window of kiosko/SQLite never had to prove under real pressure: a central system that knows, unambiguously, which of a hundred simultaneous updates arrived "first" according to the official record, and that rejects, with a clear "retry" signal, anyone who arrived late.

The four production catalogs

REST Catalog — the protocol, not a product

The REST Catalog isn't a specific catalog — it's an open specification, published by the Apache Iceberg project itself, for how any engine should talk to any catalog over HTTP: a standard set of endpoints (GET /v1/namespaces/{ns}/tables/{table}, among others) and JSON payloads, documented with OpenAPI. The central idea: instead of every engine (Spark, Trino, PyIceberg, DuckDB…) needing a different client for each type of catalog (Hive Metastore, Glue, SQL…), an engine implements the REST protocol once, and from there it can talk to any catalog that also implements that same protocol — including Unity Catalog and Polaris, the next two on this list.

The conditional atomic update, in this protocol, works like this: the client sends the proposed changes along with the metadata.json it expected to find current; the REST server validates that expectation against its own record, and only if it matches, accepts the change and moves the pointer. If it doesn't match — another client already wrote first — the server responds with an explicit conflict error, and the client decides whether to retry.

PyIceberg already ships with native support for this protocol, verified in your own installation:

from pyiceberg.catalog import AVAILABLE_CATALOGS
print(AVAILABLE_CATALOGS)

{<CatalogType.REST: 'rest'>: ..., <CatalogType.HIVE: 'hive'>: ...,
 <CatalogType.GLUE: 'glue'>: ..., <CatalogType.DYNAMODB: 'dynamodb'>: ...,
 <CatalogType.SQL: 'sql'>: ..., <CatalogType.IN_MEMORY: 'in-memory'>: ...,
 <CatalogType.BIGQUERY: 'bigquery'>: ...}

Connecting to a real REST catalog would be, in code, as simple as changing two arguments on the line you already know from module 1:

# Does NOT run in this guide -- needs a real REST server and credentials.
# Same load_catalog() as always, type="rest" instead of type="sql".
catalog = load_catalog("kiosko", type="rest", uri="https://your-rest-server/", ...)

That line deliberately doesn't run in this guide: you'd need a real REST server running somewhere, with its own credentials — exactly the boundary this lesson respects.

AWS Glue Data Catalog — AWS's managed metastore

AWS Glue Data Catalog is an AWS managed service that acts as the central metastore for an account's whole data ecosystem — it wasn't born thinking only about Iceberg, but it's natively supported it for several years. When an Iceberg table uses Glue as its catalog, Glue stores the current metadata.json's location as a parameter of its own table registry, and uses internal locking mechanisms (with a conditional atomic update backed by DynamoDB) to solve the same "a single pointer, updated unambiguously" guarantee you already know.

Glue's concrete advantage over standing up your own REST server: it's serverless — there's no process you have to operate — it integrates directly with IAM (the same permissions already managing the rest of your AWS account), and it's, in practice, the default catalog for any team that already lives inside AWS. The stated downside: accessing Glue from outside AWS requires configuring explicit IAM credentials, and the service has request-rate limits that can become a bottleneck with many concurrent writers.

Unity Catalog — governance + Iceberg, with a bridge to Delta

Unity Catalog, from Databricks, started as a data governance catalog — permissions, lineage, auditing — for Delta tables. In 2026 it natively implements Iceberg's REST Catalog protocol: it supports managed Iceberg tables (created, read, written, and optimized directly inside Unity Catalog) with full read/write access for external Iceberg clients — Spark, Trino, DuckDB, or any engine that speaks the REST protocol — with credential vending: instead of handing you long-lived credentials, the catalog generates temporary, minimally scoped credentials for each specific operation.

The piece connecting this lesson to this same module's lesson 7: Unity Catalog also supports Delta UniForm, a Delta Lake feature that generates, alongside every Delta table, Iceberg-compatible metadata on the same Parquet files — so a native Delta table can be read, with no conversion, from a client that only knows how to speak Iceberg. It isn't a coincidence that Databricks's governance catalog ends up implementing its historic competitor's open protocol: it is, with concrete evidence, the same convergence lesson 7 develops in depth.

Apache Polaris — REST + access control, donated to Apache

Apache Polaris, originated at Snowflake and donated as an open-source project to the Apache Software Foundation, is a REST catalog server designed specifically for multi-engine and multi-cloud coordination. It implements the REST Catalog protocol as its base, and adds fine-grained role-based access control (RBAC) and its own credential vending: when an engine requests access to a table, Polaris contacts the corresponding cloud provider's security token service (STS) to generate short-lived, narrowly scoped storage credentials — it never hands out a master key.

The difference in approach from Glue or Unity Catalog: Polaris isn't tied to a single cloud provider or a single compute engine — its explicit goal is to be the neutral catalog any combination of engines (Spark on one side, Snowflake on another, Trino on a third) can share without any of them having to "own" the catalog infrastructure.

Diagram: the same guarantee, four implementations

flowchart TB
    G["The shared guarantee:\na single pointer to the current metadata.json,\nconditional atomic update"]

    G --> K["kiosko / SQLite\n(this guide, M1-M6, M7-M8)\n1 process, no real concurrency"]
    G --> R["REST Catalog\nopen protocol, HTTP,\nany compatible engine"]
    G --> GL["AWS Glue Data Catalog\nAWS's serverless metastore,\nIAM + DynamoDB"]
    G --> U["Unity Catalog\nREST + governance + credential vending,\nDelta UniForm bridge"]
    G --> P["Apache Polaris\nREST + RBAC + credential vending,\nmulti-cloud, multi-engine"]

Going deeper: why this guide never connects to any of the four

It would be technically possible, with an AWS account or a Databricks workspace, to adapt any code from this guide to one of these four catalogs by changing only load_catalog()'s type= argument — the PyIceberg API you already know (create_namespace(), create_table(), table.append(), table.scan()) doesn't change depending on which catalog you use underneath. But this guide stated, from its design, an explicit boundary: managed catalogs with a real account and credentials are aws-core-services-guide's territory, not this guide's. The reason isn't only about scope — it's that really verifying these four connections would require, for each one, an active account, correctly configured credentials, and (for Glue and Unity Catalog) a real infrastructure cost, however minimal. This lesson prefers to name each guarantee precisely and quote its official documentation, instead of faking a connection that wasn't really verified — the same discipline you already saw in module 6 with Spark's MERGE INTO.

Common mistakes

Thinking "REST catalog" means "one specific catalog called REST." What happens: someone looks for "REST catalog" expecting to find an installable product with that exact name, and gets confused discovering that Unity Catalog, Polaris, and even Glue (with an additional layer) can speak "REST." Why it happens: the name sounds like a product, but it's a protocol. How to spot it: if your confusion is "which of the four IS the REST catalog?", the question is framed wrong. How to fix it: REST Catalog is the specification — the set of HTTP endpoints and JSON payloads; Unity Catalog and Polaris are implementations of that specification (among others that exist in the ecosystem, like the Iceberg project's own reference REST catalog, or Project Nessie); Glue has its own native protocol, distinct from REST, though in 2026 many engines can also talk to Glue through a REST compatibility layer.

Assuming a production catalog solves, on its own, data access control. What happens: someone sets up a REST or Glue catalog, confirms the metadata pointer works, and assumes they already have a complete permission system for who can read or write each table. Why it happens: the catalog does control who can update the pointer — that's its central guarantee — but that's different from a data governance system with roles, masked sensitive columns, or access auditing. How to spot it: if your question is "who can see dim_product's unit_cost column?", you've already left "what the catalog guarantees" behind and entered data governance territory. How to fix it: Unity Catalog and Polaris do include fine-grained access control as part of their offering — that's why this lesson names them explicitly — but a "pure" REST catalog or Glue with no additional layers don't include it automatically; that topic belongs to data-reliability-and-governance-guide, not this lesson.

Exercises

Exercise 1 — Classify the four catalogs along two axes: "open vs. proprietary protocol" and "provider-managed vs. self-hosted." With what you read in this lesson, place REST Catalog, Glue, Unity Catalog, and Polaris on those two axes.

See solution

Protocol: REST Catalog is, by definition, open — it's the specification itself; Unity Catalog and Polaris implement that open protocol (plus their own additional layers); Glue has its own native AWS-specific protocol, though with growing REST compatibility in the ecosystem. Hosting: Glue and Unity Catalog are provider-managed services (AWS and Databricks respectively) — you don't operate the server; Polaris, though it originated at Snowflake, is open source and can be self-hosted on your own infrastructure; a "generic" REST server can also be self-hosted. The practical conclusion: if your team already lives inside AWS, Glue is the lowest-friction option; if you need multi-cloud or multi-engine with no dependence on a single provider, self-hosted Polaris or a generic REST catalog are the more neutral options.

Exercise 2 — Explain why the conditional atomic update (compare-and-swap) matters more in a production catalog than in kiosko/SQLite. Think about how many different processes wrote to kiosko.dim_product throughout this guide.

See solution

Throughout modules 1 through 6, exactly one process at a time wrote to kiosko.dim_product — your own script, run sequentially, never two scripts competing for the same commit at the same time. SQLite gave you the conditional atomic update guarantee, but it never got tested under real concurrency, because there was never a second writer competing. In a production catalog, with Spark, a notebook, and an ingestion pipeline writing simultaneously to the same table, that guarantee stops being theoretical: without it, two commits arriving almost at the same time could silently overwrite each other — one of the two would win with the catalog never detecting the conflict — corrupting the snapshot history that makes module 3's time travel possible. The guarantee was always there; this lesson is the first time this guide explains why it really matters.

Exercise 3 — Prediction: if aws-core-services-guide connected this same kiosko.dim_product table to a real AWS Glue Data Catalog, which line of code would change, and which would stay identical? Use the load_catalog() line you already know from module 1.

See solution

The load_catalog() call would change — type="sql" with uri="sqlite:///..." and warehouse="file://..." would get replaced by type="glue" with the corresponding AWS credentials (region, and typically a warehouse pointing at an S3 bucket instead of a local path). Everything else would stay identical: catalog.create_namespace("kiosko"), catalog.create_table("kiosko.dim_product", schema=...), table.append(...), table.overwrite(...), table.scan(snapshot_id=...), table.maintenance.expire_snapshots() — every line of code from this guide's modules 1 through 7 would work with no changes against a real Glue catalog. This is, precisely, why Iceberg separates "table format" from "catalog": the code that talks to the table doesn't need to know, and doesn't care, which of this lesson's five catalogs (kiosko included) is resolving the pointer underneath.

Summary and next step

In this lesson you pinned down the minimum guarantee any Iceberg catalog meets — a single pointer to the current metadata, updated atomically and conditionally — the same one kiosko/SQLite gave you without you noticing since module 2. You learned the four production catalogs that sustain that guarantee at real scale: REST Catalog (the open protocol), AWS Glue Data Catalog (AWS's managed metastore), Unity Catalog (governance + Iceberg + bridge to Delta), and Apache Polaris (REST + access control, multi-cloud). None of the four got really connected — that connection, with a real account and credentials, belongs to aws-core-services-guide.

Before moving on you should be able to: explain the conditional atomic update in your own words; and name, for each of the four catalogs, what sets it apart from the other three.

Lesson 3 finally returns to code that does run: it rebuilds kiosko.dim_product with the exact state module 3 left it in, adds five nights of a pipeline that repeats itself unnecessarily, and measures, with real table.history(), how much that costs.

Resources

• Apache Iceberg — REST Catalog Open API Specification, the formal source for the protocol Unity Catalog and Polaris implement. iceberg.apache.org/rest-catalog-spec. In English.
• PyIceberg — API reference, the Catalog section, with the full list of supported catalog types (rest, glue, sql, hive, dynamodb, bigquery, in-memory), verified in this guide's installation. py.iceberg.apache.org/api. In English.
• Databricks — official documentation, "What is Apache Iceberg in Databricks," source for the Iceberg tables managed by Unity Catalog and its REST Catalog protocol support. docs.databricks.com/aws/en/iceberg. In English.
• This guide's DESIGN doc — module 7's section, and the explicit boundary with aws-core-services-guide this lesson respects. src/guides/lakehouse-and-iceberg-guide/DISENO.md. In Spanish.