Module 6: Incremental Models And Idempotency

`--full-refresh` when you need to rebuild

Description

An incremental model gains efficiency in exchange for a real limitation: every normal run only sees, and can only fix, the partition you asked for with run_date. If the model's logic changed — for example, if someone fixed a bug in the revenue calculation, or added a new column —, that change would never apply to partitions already built with the old logic, only to the ones you process from then on. --full-refresh is the flag that solves exactly that scenario: it tells dbt to discard the destination table completely and rebuild it from scratch, with the current SELECT, with no run_date filter involved at all.

Connection to the module. Lesson 6 proved that two normal runs, over the same date, don't duplicate anything. This lesson presents the third operational piece every incremental model needs: how to rebuild it whole, on purpose, when partition-by-partition stops being enough.

An analogy: rewriting the whole diary, on purpose

Lesson 1 used the image of a diary you add to page by page, without rewriting the whole notebook every night. --full-refresh is the day you do decide to rewrite the complete notebook — not because the "add one page at a time" method is broken, but because you discovered every earlier page has a formatting mistake you want to fix all at once: you changed the date style you use in each entry's header, and you want the old pages to reflect it too, not just the new ones. Rewriting everything, in that case, is the right call — but it's a deliberate, occasional event, not the nightly habit.

Worked example: rebuilding fact_orders from scratch

With fact_orders already run twice over run_date: "2026-08-05" (lessons 5 and 6, 40 rows confirmed), run:

dbt run --select fact_orders --full-refresh

What to expect.

Running with dbt=1.12.2
Registered adapter: duckdb=1.11.0
Found 7 models, 15 data tests, 1 snapshot, 4 sources, 501 macros

Concurrency: 4 threads (target='dev')

1 of 1 START sql incremental model main.fact_orders ............................ [RUN]
1 of 1 OK created sql incremental model main.fact_orders ....................... [OK in 0.10s]

Finished running 1 incremental model in 0 hours 0 minutes and 0.17 seconds (0.17s).

Completed successfully

Done. PASS=1 WARN=0 ERROR=0 SKIP=0 NO-OP=0 REUSED=0 TOTAL=1

Notice something important that confirms lesson 3, now on its fourth condition: you didn't need to pass --vars. Without --full-refresh, running fact_orders with no run_date produces the error you already saw in lesson 5 (Required var 'run_date' not found); with --full-refresh, that same command runs with no problem at all, because is_incremental() returns False as soon as it sees the flag — the {% if is_incremental() %} block containing the only use of var("run_date") never gets evaluated.

Inspecting the SQL: no filter, complete table

Confirm, with the same technique from lesson 3, what SQL dbt actually ran:

cat target/run/kiosko_analytics/models/marts/fact_orders.sql

What to expect (shortened; dbt-duckdb first builds a temporary table before replacing the definitive one).

create table "kiosko"."main"."fact_orders__dbt_tmp" as (
    select
        o.order_id,
        o.store_id,
        o.product_id,
        o.quantity,
        o.unit_price,
        o.quantity * o.unit_price as revenue,
        o.order_ts
    from "kiosko"."main"."stg_orders" as o
    inner join "kiosko"."main"."dim_store" as ds
        on o.store_id = ds.store_id
    inner join "kiosko"."main"."dim_date" as dd
        on cast(o.order_ts as date) = dd.calendar_date
);

No WHERE, no DELETE, no incremental INSERT at all — it's, structurally, the same kind of operation you already saw in module 3, back when fact_orders was still a plain table: a complete SELECT, with no conditional filter. The visible difference compared with a normal run (lesson 5, lesson 6) is exactly that absence of WHERE cast(o.order_ts as date) = cast('...' as date) — the proof, at the level of generated SQL, that --full-refresh disabled the is_incremental() block entirely.

Confirming the result is identical

dbt show --inline "select count(*) as n_rows, sum(revenue) as total_revenue from {{ ref('fact_orders') }}"

What to expect.

Previewing inline node:
| n_rows | total_revenue |
| ------ | -------------- |
|     40 |         106.15 |

Same numbers as always. Confirm the per-date breakdown too — the same check you already used in lesson 6:

dbt show --inline "select cast(order_ts as date) as order_date, count(*) as n from {{ ref('fact_orders') }} group by 1 order by 1" --limit 10

What to expect.

Previewing inline node:
| order_date |  n |
| ---------- | -- |
| 2026-08-03 |  8 |
| 2026-08-04 |  6 |
| 2026-08-05 |  2 |
| 2026-08-06 |  5 |
| 2026-08-07 |  7 |
| 2026-08-08 |  9 |
| 2026-08-09 |  3 |

Identical, day by day, to what you already confirmed in lesson 6 with incremental runs. This is the result a well-designed model is expected to produce: rebuilding everything from scratch produces exactly the same content as rebuilding it partition by partition — the only real difference is how much work dbt did to get there, not the final result.

When to actually use --full-refresh

Three concrete scenarios, all relevant well beyond this practice project:

  • You changed the SELECT's logic. If you fixed, say, a bug in the revenue calculation (imagine it originally said quantity + unit_price instead of quantity * unit_price — a real mistake you'd only notice by looking at the final number), normal incremental runs, from then on, would only apply the fix to the partitions you explicitly reprocess. The old partitions would keep the incorrect calculation until you run --full-refresh once, so the whole table reflects the corrected logic.
  • You changed the strategy or the unique_key. Switching from append to delete+insert (or changing which column is unique_key) over a table that already has data built with the old configuration can leave the table in a state that's inconsistent with the new configuration. --full-refresh guarantees the complete table gets rebuilt under the new rules, from scratch.
  • You suspect the table got corrupted. If, for any reason outside dbt (a run interrupted halfway, an accidental manual edit), the destination table ended up in a state you don't trust, --full-refresh is the simplest way to get back to a known, correct state, with no need to diagnose exactly what went wrong.

What the three scenarios have in common: --full-refresh is a deliberate, occasional operation, triggered for a specific reason — never the default command you run "just to be safe" on every normal run, because doing so would completely cancel out the work savings that motivated this entire module (lesson 2).

Diagram: an incremental run's two paths

flowchart TD
    A["dbt run --select fact_orders"] --> B{"--full-refresh\npresent?"}
    B -- Yes --> C["is_incremental() = False\ncomplete SELECT, no filter\ndoesn't require --vars"]
    B -- No --> D{"Does the table\nalready exist?"}
    D -- No --> C
    D -- Yes --> E["is_incremental() = True\nrequires --vars run_date\nDELETE + INSERT only that partition"]

Common mistakes

Using --full-refresh on every run, "to not have to think about run_date". What happens: someone, tired of having to pass --vars every time, adopts the habit of always running dbt run --select fact_orders --full-refresh, avoiding the "var not found" error that way. Why it happens: --full-refresh does solve the immediate problem (the error goes away), so it feels like a reasonable shortcut. How to spot it: in a toy-data project like Kiosko, there's no visible sign anything's wrong — the result is the same; the problem only shows up at real scale, where rebuilding the complete table on every run is, precisely, the cost this entire module exists to avoid (lesson 2). How to fix it: --full-refresh is for deliberate, occasional rebuilds, not the routine command — the normal, day-to-day flow is dbt run --select fact_orders --vars '{"run_date": "..."}', with the date matching that run's new data.

Thinking --full-refresh drops and rebuilds the whole project, not just the selected model. What happens: someone runs dbt run --full-refresh with no --select at all, expecting it to only affect fact_orders, and is surprised that other incremental models in the project (if there were any) also get fully rebuilt. Why it happens: it's easy to forget that --full-refresh is a flag that changes the behavior of every incremental model the command touches, not a setting specific to a single model. How to spot it: always check which models your --select includes before adding --full-refresh — in this project, with only one incremental model (fact_orders), the difference isn't visible, but in a project with several incremental models it would be. How to fix it: always combine --full-refresh with an explicit --select when you want to limit the rebuild to a specific model, exactly as this lesson's worked example does (--select fact_orders --full-refresh).

Expecting --full-refresh to be faster than a normal incremental run. What happens: someone, after seeing that --full-refresh doesn't require --vars, assumes it's also faster or simpler to run in general. Why it happens: fewer arguments in the command intuitively feels like "less work." How to spot it: with Kiosko's toy data the time difference is undetectable, but conceptually it's the opposite — --full-refresh always processes every source row, while a normal incremental run processes only the run_date partition; at real scale, --full-refresh is, almost always, the more expensive of the two operations. How to fix it: remember lesson 2's full argument — an incremental model's value is avoiding --full-refresh's work on every routine run, reserving it for this lesson's three specific scenarios.

Exercises

Exercise 1 — Confirm --full-refresh works even when you also pass --vars. Run dbt run --select fact_orders --full-refresh --vars '{"run_date": "2026-08-05"}' — with both the flag and the variable at once. Does the command fail, or does run_date simply get ignored?

See solution

The command runs with no problem, finishing with PASS=1 and 40 rows — run_date doesn't cause any error, much less affect the result. The reason is the same one this lesson already explained: with --full-refresh present, is_incremental() returns False no matter what else, so the {% if is_incremental() %} block — the only place in the file where var("run_date") is used — never gets evaluated. Passing --vars alongside --full-refresh isn't an error, it's simply a variable dbt receives but that the model never ends up needing in that execution mode.

Exercise 2 — Simulate the "I fixed the model's logic" scenario. Temporarily change fact_orders.sql's revenue calculation from o.quantity * o.unit_price to o.quantity * o.unit_price * 1.0 (a change that doesn't alter the numeric value, but simulates "fixing the logic"). Run dbt run --select fact_orders --vars '{"run_date": "2026-08-05"}' first (a normal run, which would only touch that partition), and then dbt run --select fact_orders --full-refresh. Which of the two runs applies the change to all 40 rows?

See solution

Only the --full-refresh run applies the change to all 40 rows — it rebuilds the whole table with the new SELECT. The normal run, scoped to run_date: "2026-08-05", would only rebuild that date's two rows with the new logic; the other 38 rows would keep the result they had from the last time they were processed, regardless of the .sql file already having the fixed logic. This is exactly the scenario the "When to actually use --full-refresh" section describes as the first of the three cases: a change to the SELECT's logic requires --full-refresh to propagate across the whole table, not just the partition you reprocess after the change. Undo the revenue change before continuing.

Exercise 3 — Argue why --full-refresh doesn't break lesson 6's idempotency guarantee. In 2-3 sentences, explain why running --full-refresh several times in a row — with no change to the SELECT between one run and the next — also always produces the same result, even though the mechanism (rebuild everything) is different from a normal incremental run's (delete+insert over a partition).

See solution

--full-refresh is, in essence, exactly what modules 3 through 5's table materialization already was: a fixed, deterministic SELECT (no random(), no CURRENT_DATE, with Kiosko's data fixed on disk) that always produces the same result over the same input data, no matter how many times it runs. --full-refresh's idempotency doesn't depend on any comparison mechanism between runs (the way delete+insert does, which needs unique_key to know what to delete) — it depends, simply, on the SELECT being a pure function of its input data, the same property that table materialization already guaranteed since module 3.

Summary and next step

This lesson presented --full-refresh: the flag that disables is_incremental() entirely, regardless of the warehouse's state, forcing dbt to rebuild the whole table from the current SELECT — no filter, no DELETE, no need for --vars. You confirmed, with the actual SQL that ran, that the absence of the WHERE is the structural difference compared with a normal run, and that the final result — 40 rows, 106.15 revenue, the same per-date breakdown — is identical to rebuilding partition by partition. You saw the three real scenarios where --full-refresh is the right tool: a change to the model's logic, a change of strategy or unique_key, or a destination table in a state you don't trust.

Before moving on you should be able to: explain why --full-refresh doesn't need --vars; and name the three scenarios where --full-refresh is the correct choice, instead of a normal incremental run.

Lesson 8 closes the module with a mini-project that brings everything together: fact_orders's conversion, the two identical runs, --full-refresh, the complete test suite, and Kiosko's project's sixth commit.

Resources