Partition transforms: `IdentityTransform`, `BucketTransform`, `DayTransform`

Description

Lessons 2 and 3 talked about "partitioning" as if there were only one way to do it: grouping rows by a column's exact value, like store_id. This lesson gives you precise vocabulary for the three ways you're going to use for the rest of this module — each one solves a different problem, and picking the wrong one has a real, measurable cost, which this lesson also shows you. IdentityTransform preserves the value as is; BucketTransform spreads it across buckets via a hash; DayTransform truncates it to its calendar date.

Connection to the module. This lesson is the last conceptual stop before the real execution. Lesson 5 is going to use IdentityTransform over store_id in kiosko.fact_orders_at_scale's initial PartitionSpec; lesson 6 is going to add DayTransform over order_ts in that spec's evolution. BucketTransform doesn't get used on this guide's main table — store_id, with only three values, doesn't need it — but it's explained here, with executed evidence, because it's the right tool for the case in Kiosko that does have high cardinality: franchise_id, with 250,000 distinct values.

An analogy: three ways to organize the same cabinet

Going back to lesson 2's cabinet: there's more than one reasonable way to decide what goes in each drawer. If you organize by store — three stores, three drawers — each drawer has a manageable size and the label is direct: "this is exactly S01's stuff." That's IdentityTransform: the drawer's label is the value itself.

Now imagine that, instead of three stores, you had 250,000 franchises, and you decided to apply the same logic — one drawer per franchise. You'd end up with 250,000 drawers, most of them nearly empty, a completely unmanageable cabinet. The reasonable solution is to group several franchises into a fixed number of larger drawers — say, 16 — using a deterministic rule to decide which one each goes into (the same franchise always lands in the same drawer, but several franchises share a drawer). That's BucketTransform: it doesn't give you a drawer per value, it gives you a fixed number of drawers, and a hash function decides which one each value gets.

And if you organized by the operation's date, you wouldn't want a drawer per exact second — you'd be back to the same problem of 250,000 nearly-empty drawers — you'd want a drawer per day. That's DayTransform: it truncates a full timestamp to its calendar date, naturally grouping everything that happened on the same day.

Worked example: the three transforms, run directly

PyIceberg lets you invoke a transform directly on a value, with no table needed — it's the most direct way to see, unambiguously, what each one produces:

# l4_transforms_demo.py
from datetime import datetime

from pyiceberg.transforms import IdentityTransform, BucketTransform, DayTransform
from pyiceberg.types import IntegerType, StringType, TimestampType

print("=== IdentityTransform: the partition value IS the column value ===")
identity = IdentityTransform()
identity_fn = identity.transform(StringType())
for store_id in ["S01", "S02", "S03"]:
    print(f"  IdentityTransform()({store_id!r}) -> {identity_fn(store_id)!r}")

print("\n=== BucketTransform(4): deterministic hash into N buckets ===")
bucket = BucketTransform(4)
bucket_fn = bucket.transform(IntegerType())
for franchise_id in [0, 1, 2, 3, 4, 100, 249_999]:
    print(f"  BucketTransform(4)(franchise_id={franchise_id}) -> bucket {bucket_fn(franchise_id)}")

print("\n=== DayTransform: truncates a timestamp to its calendar date ===")
day = DayTransform()
day_fn = day.transform(TimestampType())
for iso in ["2026-08-03T08:14:00", "2026-08-03T23:59:00", "2026-08-09T10:05:00"]:
    ts = datetime.fromisoformat(iso)
    epoch_days = day_fn(ts)
    print(f"  DayTransform()({iso}) -> {epoch_days} days since epoch (1970-01-01)")

What to expect (verified by running the real script):

=== IdentityTransform: the partition value IS the column value ===
  IdentityTransform()('S01') -> 'S01'
  IdentityTransform()('S02') -> 'S02'
  IdentityTransform()('S03') -> 'S03'

=== BucketTransform(4): deterministic hash into N buckets ===
  BucketTransform(4)(franchise_id=0) -> bucket 0
  BucketTransform(4)(franchise_id=1) -> bucket 0
  BucketTransform(4)(franchise_id=2) -> bucket 0
  BucketTransform(4)(franchise_id=3) -> bucket 3
  BucketTransform(4)(franchise_id=4) -> bucket 2
  BucketTransform(4)(franchise_id=100) -> bucket 0
  BucketTransform(4)(franchise_id=249999) -> bucket 0

=== DayTransform: truncates a timestamp to its calendar date ===
  DayTransform()(2026-08-03T08:14:00) -> 20668 days since epoch (1970-01-01)
  DayTransform()(2026-08-03T23:59:00) -> 20668 days since epoch (1970-01-01)
  DayTransform()(2026-08-09T10:05:00) -> 20674 days since epoch (1970-01-01)

Three direct observations from this output. First, IdentityTransform is literally the identity function: the input and output value are the same object, with no real transformation at all — the name isn't a coincidence. Second, BucketTransform(4) produces integers between 0 and 3 — four buckets, as requested — and the assignment is deterministic but not intuitive at a glance: franchise_id=0, 1, 2, and 100 all land in bucket 0, while 3 lands in 3 and 4 in 2 — there's no visible pattern to the eye, because the hash is deliberately designed to distribute evenly regardless of the order of the input values. Third, DayTransform collapses 08:14:00 and 23:59:00 of the same day (2026-08-03) to the same value — 20668 — while a different date (2026-08-09) produces a different value (20674); the number itself is the count of days since 1970-01-01 (the Unix epoch), the internal representation Iceberg uses — when you query a table partitioned by DayTransform with table.inspect.partitions() (lesson 7), you're going to see that same value already converted to a readable date (datetime.date(2026, 8, 3)), not the raw integer.

When to use each one

Transform	Use it when...	The risk of NOT using it
IdentityTransform	The column has few distinct values, and you usually filter by its exact value (store_id, with 3 values)	None special — it's the default choice for low-cardinality columns
BucketTransform(N)	The column has high cardinality (franchise_id, with 250,000 values), and you want a fixed, manageable number of files	Using IdentityTransform on a high-cardinality column produces exploding partitioning: an almost-empty file per distinct value — this lesson's common mistake
DayTransform	You often filter by date ranges (order_ts >= '2026-08-05'), or you need to expire old data by age	Without truncating the date, every exact timestamp would be its own partition — the same explosion problem, applied to time

Diagram: from the column to the PartitionSpec

flowchart LR
    subgraph valores["Business values"]
        S["store_id: S01, S02, S03\n(3 values)"]
        F["franchise_id: 0..249,999\n(250,000 values)"]
        T["order_ts: exact timestamps\n(potentially infinite values)"]
    end

    S -->|"IdentityTransform"| PS["PartitionSpec\nkiosko.fact_orders_at_scale\n(lesson 5)"]
    T -->|"DayTransform -> order_day"| PS
    F -.->|"BucketTransform(N)\n(not used on this guide's\nmain table,\nsee Exercise 2)"| BX["N manageable buckets"]

Going deeper: why partition field_ids start at 1000

Notice a detail you're going to see again in lesson 5: when you build a PartitionField by hand, its field_id is going to start at 1000, not at 1. This isn't arbitrary — it's the same convention documented in PyIceberg's official examples, and it exists for the same reason you already saw in module 4 with the schema's field_ids: every internal numeric identifier in Iceberg needs to be unique and stable forever, even after a column gets dropped or a partition field gets removed. Starting partition field_ids at 1000 leaves room — the numbers 1 through 999 — for business schema columns (like the ones you already saw in modules 1 through 4, with field_ids 1 through 7), so both numbering spaces never collide with each other.

It's also worth noting the difference between how you're going to create a spec for the first time (lesson 5) and how you're going to evolve it afterward (lesson 6). To create the initial spec for a table that doesn't yet exist, there's no live schema to ask "what's store_id's field_id?" — that's why you have to build the PartitionField by hand, with that column's exact numeric source_id in the Schema. Once the table already exists, table.update_spec().add_field("order_ts", DayTransform(), "order_day") lets you refer to the column by name — because now there really is a live schema, with a real catalog, that can resolve that name for you.

Common mistakes

Using IdentityTransform on a high-cardinality column, "because it's the simplest option." What happens: someone, without thinking twice, partitions kiosko.fact_orders_at_scale by franchise_id with IdentityTransform, instead of store_id. Why it happens: IdentityTransform is, conceptually, the easiest transform to understand — "the partition is the value" — so it's the default choice if the column's cardinality isn't known in advance. How to spot it: if your table ends up with more than a few thousand data files after a load, and each one weighs a few kilobytes, suspect exploding partitioning — check how many distinct values the column you partitioned by has. How to fix it: for franchise_id, with 250,000 possible values, the correct choice is BucketTransform(N) with a reasonable N (tens, not hundreds of thousands) — it groups many franchises into each bucket, instead of an almost-empty drawer per franchise. This lesson's Exercise 2 has you calculate that N with evidence.

Expecting the same input value to always land in the same bucket number, no matter how many total buckets you choose. What happens: someone runs BucketTransform(4)(franchise_id=100) and gets 0, then runs BucketTransform(16)(franchise_id=100) and is surprised to get a different number. Why it happens: it's easy to assume a value's hash is a fixed property of the value itself, independent of how many buckets exist. How to spot it: if your code assumes a value's bucket number doesn't change when N changes, revisit this lesson's worked example again — the result depends both on the input value and on the total number of buckets (N), because the hash gets reduced modulo N. How to fix it: treat a BucketTransform's number of buckets as a decision that, once made and with data already written, you shouldn't change lightly — changing it is, formally, a partition evolution (lesson 6), and old files are going to stay organized under the old N while new ones use the new N, exactly like you're going to see with DayTransform in lesson 7.

Exercises

Exercise 1 — Reproduce the three transforms yourself, with different values. Run this lesson's worked example, but with store_id replaced by product names (P001 through P004), franchise_id with at least ten new values of your choosing, and three dates of your choosing for DayTransform. Confirm the pattern holds: IdentityTransform changes nothing, BucketTransform spreads with no visible pattern, DayTransform collapses hours from the same day.

See solution

There's no single "correct" output — it depends on the values you choose — but the structural pattern should always hold: every IdentityTransform value should be identical to its input; BucketTransform's values should always fall between 0 and N-1 (with N the number of buckets you use), with no visible order related to the input value; and any pair of timestamps from the same calendar day should produce the same integer under DayTransform, while a timestamp from a different day should produce a different integer.

Exercise 2 — Calculate a reasonable BucketTransform N for franchise_id. With 250,000 franchises, and knowing an overly small Parquet file (a few KB) wastes metadata overhead, and an overly large file makes pruning harder, what range of N seems reasonable to you for BucketTransform(N) over franchise_id? Justify it with a simple calculation of how many franchises would land, on average, in each bucket.

See solution

With 250,000 franchises distributed evenly among N buckets, each bucket receives, on average, 250,000 / N franchises. With N=16 (a common value in official documentation examples), that's approximately 15,625 franchises per bucket — each with 40 rows, so each bucket would end up with around 625,000 rows, a reasonable file size for Parquet. With too small an N (say, 4), each bucket would carry proportionally more rows — larger files, less possible parallelism when reading; with too large an N (say, 100,000), you'd be back to approaching the exploding-partitioning problem this transform exists to avoid. There's no single "correct" N — it's an engineering decision that depends on the real volume and the expected query patterns, not a fixed formula.

Exercise 3 — Prediction: why does DayTransform truncate to day, and not offer, say, SecondTransform? Without looking it up yet, predict: why do you think Iceberg documents time transforms down to the hour (HourTransform) but nothing finer than that? Think about this module's same cardinality problem.

See solution

A transform finer than the hour — by second, or by millisecond — would reproduce exactly the same exploding-partitioning problem you already saw with IdentityTransform over franchise_id: with timestamps that in practice almost never repeat at the exact same second, every partition would end up with one or very few rows, multiplying the number of files with no real pruning benefit. The time transforms Iceberg does offer — YearTransform, MonthTransform, DayTransform, HourTransform — are each designed for the granularity level at which business queries typically filter ("give me this month's data," "give me today's data"), not for the maximum technical resolution the timestamp type is capable of storing.

Summary and next step

In this lesson you learned, with executed evidence, the three partition transforms you're going to use for the rest of this module: IdentityTransform (the value as is, for low-cardinality columns like store_id), BucketTransform (deterministic hash into N buckets, for high-cardinality columns like franchise_id), and DayTransform (calendar date, for time columns like order_ts). You also saw why a partition field's field_id starts at 1000, and the difference between building a spec by hand (new table) and evolving it by name (existing table).

Before moving on you should be able to: explain in your own words when to use each of the three transforms; and anticipate why IdentityTransform over franchise_id would be a mistake, even though it technically works without throwing any error.

Lesson 5 uses the first of the three, IdentityTransform, to create kiosko.fact_orders_at_scale's real PartitionSpec — the full, full-scale execution of the contrast this module has promised since lesson 1.

Resources

• Apache Iceberg — official documentation, "Partitioning," "Partition Transforms" section (the full list of available transforms, including the ones this lesson doesn't use: year, month, hour, truncate). iceberg.apache.org/docs/latest/partitioning. In English.
• PyIceberg — API reference, IdentityTransform, BucketTransform, DayTransform, and the rest of the pyiceberg.transforms module. py.iceberg.apache.org/api. In English.
• Apache Iceberg — table specification ("Table Spec"), "Partitioning" section (the field_id convention starting at 1000 for partition fields). iceberg.apache.org/spec. In English.
• This guide's DESIGN doc — module 5's section, with the three transforms explicitly named as part of the scope. src/guides/lakehouse-and-iceberg-guide/DISENO.md. In Spanish.