Installing Pandera and building the DuckDB → Polars bridge

Description

This lesson actually installs the tool lesson 3 chose, and builds the technical bridge the rest of this module is going to use without explaining it again: orders_2026-08-14.csv goes into kiosko.duckdb as a real SQL table, and comes out of it as a Polars DataFrame, the only format Pandera needs to work in this guide. Not a single line of this bridge uses pandas — it's a hard rule of this entire ecosystem, not a preference of this lesson.

Connection to the module. This lesson is the hinge between the decision (lesson 3) and the build (lessons 5 through 8): everything that follows in this module — and much of the rest of this guide — starts from the same orders_s04 table and the same con.sql(...).pl() pattern this lesson builds for the first time.

The running case: orders_2026-08-14.csv enters Kiosko's warehouse

Until now, in this guide's module 1, orders_2026-08-14.csv was always read with csv.DictReader() — a list of Python dictionaries, with no SQL engine involved. This lesson reads it a different way: as a table inside kiosko.duckdb, the same warehouse file data-modeling-for-analytics-guide built and dbt-analytics-engineering-guide versioned. This guide doesn't rebuild that warehouse from scratch — that's not its job —, but it does use the same engine, DuckDB, to land S04's new file as a real SQL table: orders_s04.

An analogy: the universal plug adapter

When you travel to a country with a different electrical standard, your phone charger doesn't change — it still expects the same kind of current it always expected. What changes is the wall socket. The universal adapter doesn't generate electricity or transform it — it just has, on one side, the exact shape that fits the outlet in the country you're in, and on the other, the exact shape your charger recognizes. Without that adapter, two perfectly functional objects — the outlet, the charger — simply can't connect to each other.

DuckDB and Polars are, in that sense, two perfectly functional objects that speak different formats: DuckDB organizes data as a SQL relation, Polars organizes it as Apache Arrow columns in memory. .pl(), the method this lesson uses, is the adapter — it doesn't transform the data's content, it only changes the shape it's organized in, so Pandera (which only recognizes Polars's "plug," never a DuckDB relation directly) can read it. And, like any real adapter, it has a concrete physical component behind it: the pyarrow library, which does the real work of translating a DuckDB relation's internal format into a Polars DataFrame's internal format.

Worked example: install, land the CSV, cross the bridge

Step 1 — install the three pieces

pip install "pandera[polars]" duckdb pyarrow

Three packages, three different roles. pandera[polars] installs Pandera and Polars at once — the [polars] extra declares polars>=0.20.0 as a dependency, so a single command covers both. duckdb is the same SQL engine the ecosystem's earlier guides already used. pyarrow is the adapter from the analogy: without it, .pl() fails with ModuleNotFoundError: No module named 'pyarrow' — DuckDB's official documentation says so explicitly: "the pyarrow library must be installed for the integration to work."

What to expect. Confirm the exact three versions this guide uses by running:

# check_versions.py
import pandera
import polars as pl
import duckdb

print(f"pandera: {pandera.__version__}")
print(f"polars: {pl.__version__}")
print(f"duckdb: {duckdb.__version__}")

pandera: 0.32.1
polars: 1.43.2
duckdb: 1.5.5

duckdb 1.5.5 is the same version you already saw run in data-modeling-for-analytics-guide (module 8). pandera 0.32.1 is the current version lesson 3 investigated, published on June 29, 2026. Your install may bring a slightly different polars version — Pandera only requires >=0.20.0 —, and that doesn't break anything that follows.

Step 2 — land orders_2026-08-14.csv as a table in kiosko.duckdb

# bridge_duckdb_to_polars.py
import duckdb

con = duckdb.connect("kiosko.duckdb")

con.execute("""
    CREATE OR REPLACE TABLE orders_s04 AS
    SELECT * FROM read_csv('orders_2026-08-14.csv', header=True,
        columns={
            'order_id': 'VARCHAR', 'store_id': 'VARCHAR', 'product_id': 'VARCHAR',
            'quantity': 'BIGINT', 'unit_price': 'DOUBLE', 'order_ts': 'TIMESTAMP'
        })
""")

row_count = con.sql("SELECT COUNT(*) FROM orders_s04").fetchone()[0]
print(f"Rows loaded into orders_s04: {row_count}\n")
print(con.sql("DESCRIBE orders_s04"))

Notice two deliberate decisions in this block. First, CREATE OR REPLACE TABLE, not a plain CREATE TABLE — if you run this script a second time without OR REPLACE, DuckDB rejects the operation with CatalogException: Catalog Error: Table with name "orders_s04" already exists!, because a table with that name already exists in kiosko.duckdb from the previous run; OR REPLACE makes this script safe to run as many times as needed, without accumulating ghost tables. Second, read_csv()'s columns={...} parameter — it explicitly declares each column's type, instead of letting DuckDB guess. This matters especially for unit_price: ORD-9503's row has that field empty in the CSV, and with the type declared as DOUBLE, DuckDB converts that empty value into NULL cleanly and predictably — exactly the behavior lessons 5 and 6 need to be able to catch that row with nullable=False.

What to expect. Running python3 bridge_duckdb_to_polars.py in the folder where you saved orders_2026-08-14.csv, the output is exactly this:

Rows loaded into orders_s04: 12

┌─────────────┬─────────────┬─────────┬─────────┬─────────┬─────────┐
│ column_name │ column_type │  null   │   key   │ default │  extra  │
│   varchar   │   varchar   │ varchar │ varchar │ varchar │ varchar │
├─────────────┼─────────────┼─────────┼─────────┼─────────┼─────────┤
│ order_id    │ VARCHAR     │ YES     │ NULL    │ NULL    │ NULL    │
│ store_id    │ VARCHAR     │ YES     │ NULL    │ NULL    │ NULL    │
│ product_id  │ VARCHAR     │ YES     │ NULL    │ NULL    │ NULL    │
│ quantity    │ BIGINT      │ YES     │ NULL    │ NULL    │ NULL    │
│ unit_price  │ DOUBLE      │ YES     │ NULL    │ NULL    │ NULL    │
│ order_ts    │ TIMESTAMP   │ YES     │ NULL    │ NULL    │ NULL    │
└─────────────┴─────────────┴─────────┴─────────┴─────────┴─────────┘

Twelve rows, the same twelve from module 1, now living as a real SQL table, with types declared column by column — DESCRIBE orders_s04 is DuckDB's equivalent of asking a table "what are you made of?", the same kind of question you already asked with validate_gold_schema() in data-modeling-for-analytics-guide.

Step 3 — cross the bridge: .pl()

# bridge_duckdb_to_polars.py -- continued
df = con.sql("SELECT * FROM orders_s04").pl()

print(f"\ntype(df): {type(df)}")
print(f"df.shape: {df.shape}")
print("\ndf.schema:")
print(df.schema)

What to expect.

type(df): <class 'polars.dataframe.frame.DataFrame'>
df.shape: (12, 6)

df.schema:
Schema({'order_id': String, 'store_id': String, 'product_id': String, 'quantity': Int64, 'unit_price': Float64, 'order_ts': Datetime(time_unit='us', time_zone=None)})

con.sql("SELECT * FROM orders_s04") builds a DuckDB relation — a query, not yet a materialized result. .pl(), chained at the end, is the complete adapter: it executes the query and delivers the result as a polars.DataFrame, with DuckDB's SQL types (VARCHAR, BIGINT, DOUBLE, TIMESTAMP) already translated into their Polars equivalents (String, Int64, Float64, Datetime). None of this step touched pandas at any point — the bridge is direct, DuckDB to Polars, exactly the hard rule that holds up this entire guide.

Diagram: the complete path, from a CSV to a Pandera schema

flowchart LR
    A["orders_2026-08-14.csv\n(12 lines)"] -->|"read_csv() with\nexplicit columns="| B["kiosko.duckdb\norders_s04 table"]
    B -->|"con.sql('SELECT * FROM\norders_s04').pl()\n(needs pyarrow)"| C["polars.DataFrame\n12 rows x 6 columns"]
    C -->|"lesson 5 onward"| D["OrdersSchema.validate(df)\n(Pandera)"]

The diagram has three arrows, and each one is a different format translation: the first (read_csv) goes from plain text to a typed SQL table; the second (.pl()) goes from a SQL relation to an in-memory DataFrame; the third (which lessons 5 through 8 build) goes from a DataFrame with no rules to a DataFrame already compared against a declared schema. None of the three arrows changes the content of the data — the twelve rows stay the same twelve rows at every step —, only the shape that content lives in changes, so the next tool can read it.

Going deeper: why DuckDB stays the source, even though Pandera works over Polars

It's worth making a point clear that's prone to confusion: Pandera doesn't replace DuckDB as the data source, and Polars doesn't replace DuckDB's role as the SQL engine. kiosko.duckdb remains, in this guide and in the ecosystem's earlier ones, the place where tables live — orders_s04 today, dim_product/dim_store starting in module 3. Polars is exclusively the intermediate format Pandera needs to be able to read a SQL query's result; you don't even need to use Polars's API beyond .pl() in this guide — that API in depth (chained filters, group_by, lazy expressions) is python-for-data-engineering-guide's territory, not this guide's. The sequence is always the same: SQL in DuckDB decides what data enters validation (a filter, a join, a whole table), and Polars is just the conduit that data travels through on its way to Pandera.

This also explains why pandas is forbidden in this guide, and it isn't just an arbitrary rule: if the bridge were DuckDB → pandas → Pandera, you'd be adding an extra dependency (pandas) to the problem, when Pandera already knows how to speak Polars's format directly with no intermediate step. Fewer steps means less surface area for errors, and a simpler rule to remember: in this guide, the only DataFrame format is Polars.

Common mistakes

Forgetting pyarrow and confusing the error with a Pandera problem. What happens: someone installs pandera[polars] and duckdb, but not pyarrow, and upon reaching .pl() gets ModuleNotFoundError: No module named 'pyarrow'. Why it happens: pandera[polars] installs Polars, but doesn't install pyarrow — it's a dependency of the DuckDB↔Polars bridge, not of Pandera itself. How to spot it: if your error mentions pyarrow and happens on the .pl() line, not on any line using pa.DataFrameModel or pa.Field, the problem is with the bridge, not your schema. How to fix it: pip install pyarrow (or, in one go from the start, this lesson's full command: pip install "pandera[polars]" duckdb pyarrow).

Running CREATE TABLE without OR REPLACE a second time. What happens: someone runs bridge_duckdb_to_polars.py once, it works perfectly, and running it again (say, after editing another part of the script) gets CatalogException: Catalog Error: Table with name "orders_s04" already exists!. Why it happens: kiosko.duckdb is a file persisted on disk — unlike an in-memory connection (duckdb.connect(), with no argument), what you created in one run is still there in the next. How to spot it: the error message names the exact table and says "already exists" — it isn't a SQL syntax or data error. How to fix it: always use CREATE OR REPLACE TABLE for any table a script in this guide might need to recreate, exactly as this lesson's example does — a useful general practice for any script that gets re-run against a persistent DuckDB file.

Importing pandera instead of pandera.polars. What happens: someone copies an example from Pandera's documentation that starts with import pandera as pa (no .polars), and later, when declaring a Field with Polars-specific syntax, gets an error that doesn't make sense at a glance. Why it happens: Pandera supports several DataFrame engines — pandas, Polars, PySpark — and each has its own submodule with a slightly different API; a plain import pandera as pa defaults to the pandas integration. How to spot it: if your code doesn't explicitly import pandera.polars, and you're passing a polars.DataFrame to a schema, check your file's first line. How to fix it: in this guide, the only correct form is import pandera.polars as pa — lesson 3's example already used it that way, and every lesson that follows is going to use it that way too.

Exercises

Exercise 1 — Trigger the pyarrow error on purpose, then fix it. If you have pyarrow installed, temporarily uninstall it (pip uninstall pyarrow -y), run bridge_duckdb_to_polars.py up to the .pl() line, and confirm the exact error message. Then reinstall it and confirm the script runs all the way through again.

ModuleNotFoundError: No module named 'pyarrow'

Exercise 2 — Load only S04's rows with a positive quantity, using SQL, before reaching Polars. Modify the con.sql(...) query to filter WHERE quantity > 0 in SQL, before converting to Polars, and count how many rows remain.

df_positive = con.sql("SELECT * FROM orders_s04 WHERE quantity > 0").pl()
print(f"Rows with quantity > 0: {df_positive.height}")

Rows with quantity > 0: 11

Exercise 3 — Explain, without looking at the diagram, why the order of the three arrows can't be reversed. In 2-3 sentences, explain why the sequence has to be CSV → DuckDB → Polars → Pandera, and not, say, CSV → Polars → DuckDB → Pandera.

Summary and next step

In this lesson you actually installed the three pieces that hold up the rest of this module — pandera[polars], duckdb, pyarrow — and confirmed their exact versions. You landed orders_2026-08-14.csv as a real table in kiosko.duckdb (orders_s04, with types declared column by column), and crossed the bridge to Polars with con.sql(...).pl(), confirming with executed evidence that the twelve rows arrive intact, with the correct types, with pandas never taking part anywhere along the way.

Before moving on you should be able to: explain what exactly pyarrow does in this bridge, and why the error its absence causes has nothing to do with Pandera; reproduce the orders_s04 table from scratch in a new kiosko.duckdb; and name the difference between pandera (plain) and pandera.polars.

You have the DataFrame ready, in the right format, with S04's real data. Lesson 5 writes, for the first time in this guide, a real Pandera DataFrameModel — and runs it against this very df.

Resources

• DuckDB — "Integration with Polars" (the exact .pl() syntax, and confirmation that pyarrow is required for the integration to work). duckdb.org/docs/lts/guides/python/polars. In English.
• DuckDB — Python Client API Reference (methods for converting a DuckDB relation to Polars/Arrow/pandas). duckdb.org/docs/current/clients/python/reference. In English.
• Pandera — official documentation, Polars integration (import pandera.polars as pa, the difference from the pandas integration). pandera.readthedocs.io. In English.
• data-modeling-for-analytics-guide, module 8 — the source of kiosko.duckdb and the duckdb.connect(...) + CREATE TABLE pattern this lesson reuses. src/guides/data-modeling-for-analytics-guide/workbook/module-08-project-kioskos-analytics-warehouse/es/. In Spanish.
• This guide's DESIGN — the complete engine decision (DuckDB as the SQL source, Polars as the only bridge, pandas forbidden). src/guides/data-reliability-and-governance-guide/DISENO.md. In Spanish.