Module 7: Macros Docs And Lineage

Mini-project: Kiosko's documented project

Next →

Description

It's time to bring this module's seven lessons together into a single, end-to-end flow. Lessons 2 and 3 extracted the revenue calculation into a reusable macro, really applied to fact_orders.sql. Lessons 4 and 5 documented every model and column, and generated manifest.json and catalog.json as real artifacts. Lesson 6 read the lineage directly from those artifacts, with no additional dbt command. Lesson 7 named, with evidence you've already lived through, the exact moment running everything by hand stops being enough. This mini-project confirms module 4's fifteen tests are still green, that fact_orders still has exactly 40 rows and 106.15 revenue, and that the complete project generates clean documentation with a single command — dbt build followed by dbt docs generate, the same two commands lesson 7 already broke on purpose, now run correctly.

And it closes with this module's new piece: Kiosko project's seventh version-control commit, on top of the six modules 1 through 6 left behind.

Connection to the module. This project introduces no new concept — it's the complete synthesis of lessons 2 through 7, run end to end over the real project, including the final confirmation that documenting a project and generating its lineage changed absolutely nothing about what modules 1 through 6 already built and proved.

An analogy: the complete inventory, now with every shelf labeled

Modules 2 through 6 already used the warehouse analogy for their own mini-projects: the first complete inventory, the second floor on proven foundations, the inspector reviewing the shift, the archivist who historizes on their own, the assembly line with a new procedure. This mini-project adds that same warehouse's final piece: every shelf, every box, every procedure already has a legible label, and there's a complete map of the warehouse — hung on the wall, generated from the inventory itself, not drawn by hand by someone who might forget a shelf — that anyone new can consult without having to ask anyone where anything is.

The material: the four files that changed in this module

If you already completed lessons 2 through 6 in order, kiosko_analytics/ already has these four files. If you're jumping straight into this mini-project, this is the complete material that needs to be in place before starting.

macros/calculate_revenue.sql, lesson 3's macro:

-- macros/calculate_revenue.sql
{% macro calculate_revenue(quantity_col, price_col) -%}
    ({{ quantity_col }} * {{ price_col }})
{%- endmacro %}

models/marts/fact_orders.sql, with the macro invocation replacing the inline expression (the only change compared with module 6):

-- models/marts/fact_orders.sql
{{
    config(
        materialized='incremental',
        incremental_strategy='delete+insert',
        unique_key='order_id'
    )
}}

select
    o.order_id,
    o.store_id,
    o.product_id,
    o.quantity,
    o.unit_price,
    {{ calculate_revenue('o.quantity', 'o.unit_price') }} as revenue,
    o.order_ts
from {{ ref('stg_orders') }} as o
inner join {{ ref('dim_store') }} as ds
    on o.store_id = ds.store_id
inner join {{ ref('dim_date') }} as dd
    on cast(o.order_ts as date) = dd.calendar_date

{% if is_incremental() %}
where cast(o.order_ts as date) = cast('{{ var("run_date") }}' as date)
{% endif %}

models/marts/_models.yml, with description: on every column of the three marts, including the docs block invocation on revenue (excerpt — you wrote the complete file in lesson 4):

      - name: revenue
        description: "{{ doc('fact_orders_revenue') }}"

And models/marts/_docs.md, lesson 4's docs block:

{% docs fact_orders_revenue %}
Order line revenue, calculated with the `calculate_revenue(quantity_col, price_col)` macro
(`macros/calculate_revenue.sql`) instead of repeating `quantity * unit_price` by hand. The numeric
result didn't change compared with this model's earlier version -- the only thing that changed is
that the calculation now lives in a single, reusable place.
{% enddocs %}

No file beyond these four, no new folder — unlike modules 4's (macros/, tests/) and 5's (snapshots/) mini-projects, this module extends two folders that already existed (macros/, models/marts/), with no structure added to the project.

Why this lesson doesn't delete kiosko.duckdb either

Just like module 5's mini-project, this one also doesn't start with rm -f kiosko.duckdb. The reason is the same: dim_product_snapshot is still the result of a sequence of runs — products_v1.csv's followed by products_v2.csv's — not just the source's current state. Rebuilding everything from scratch, at this point in the guide, would lose that history with no warning at all. This mini-project works over the warehouse that already exists, exactly as it was left at the end of lesson 7.

The reference solution, verified

Part 1 — dbt build, with the correct operational contract

Lesson 7 already showed you what happens if you forget --vars. Run the command the correct way, with the matching date:

dbt build --vars '{"run_date": "2026-08-09"}'

What to expect.

Running with dbt=1.12.2
Registered adapter: duckdb=1.11.0
Found 7 models, 15 data tests, 1 snapshot, 4 sources, 502 macros

Concurrency: 4 threads (target='dev')

[... sources, staging, snapshot, and marts, in the same DAG order as module 6 ...]

16 of 23 START sql incremental model main.fact_orders .......................... [RUN]
16 of 23 OK created sql incremental model main.fact_orders ..................... [OK in 0.05s]

[... 15 data tests, all PASS ...]

Finished running 1 incremental model, 1 snapshot, 2 table models, 15 data tests, 4 view models in 0 hours 0 minutes and 0.48 seconds (0.48s).

Completed successfully

Done. PASS=23 WARN=0 ERROR=0 SKIP=0 NO-OP=0 REUSED=0 TOTAL=23

23 resources, 23 PASS, zero errors — the same Found ... 502 macros from lesson 3, one more than the 501 from modules 4 through 6. If this part fails, don't move on to Part 2 — go back to the lesson matching whichever specific resource is failing: macros/calculate_revenue.sql (lesson 3), models/marts/_models.yml or _docs.md (lesson 4).

Part 2 — Confirming the result didn't change

dbt show --inline "select count(*) as n_rows, sum(revenue) as total_revenue from {{ ref('fact_orders') }}"

What to expect.

Previewing inline node:
| n_rows | total_revenue |
| ------ | -------------- |
|     40 |         106.15 |

40 rows, 106.15 revenue — the exact same number since module 3, now calculated by calculate_revenue instead of an inline expression. Also confirm fact_orders didn't lose any relationship with the products snapshot — a check module 5's mini-project already made, and that's still true:

dbt ls --select dim_product_snapshot fact_orders

What to expect.

kiosko_analytics.dim_product_snapshot
kiosko_analytics.marts.fact_orders

Two resources, with no dependency relationship between them — exactly the same as in module 5.

Part 3 — Generating documentation and lineage

dbt docs generate --vars '{"run_date": "2026-08-09"}'

What to expect.

Running with dbt=1.12.2
Registered adapter: duckdb=1.11.0
Found 7 models, 15 data tests, 1 snapshot, 4 sources, 502 macros

Concurrency: 4 threads (target='dev')

Building catalog
Catalog written to /path/to/your/kiosko_analytics/target/catalog.json

Confirm both artifacts exist:

ls target/manifest.json target/catalog.json

What to expect. Both files, listed with no error.

Part 4 — Inspecting the manifest: the macro, the docs block, the lineage

A single Python check that brings together this module's three new pieces:

import json

manifest = json.load(open("target/manifest.json"))
fact_orders = manifest["nodes"]["model.kiosko_analytics.fact_orders"]

print("Macros fact_orders invokes:", fact_orders["depends_on"]["macros"])
print()
print("Revenue column's description:")
print(fact_orders["columns"]["revenue"]["description"])
print()
print("Model dependencies (the lineage):", fact_orders["depends_on"]["nodes"])

What to expect.

Macros fact_orders invokes: ['macro.kiosko_analytics.calculate_revenue', 'macro.dbt.is_incremental']

Revenue column's description:
Order line revenue, calculated with the `calculate_revenue(quantity_col, price_col)` macro
(`macros/calculate_revenue.sql`) instead of repeating `quantity * unit_price` by hand. The numeric
result didn't change compared with this model's earlier version -- the only thing that changed is
that the calculation now lives in a single, reusable place.

Model dependencies (the lineage): ['model.kiosko_analytics.stg_orders', 'model.kiosko_analytics.dim_store', 'model.kiosko_analytics.dim_date']

This module's three pieces, confirmed in a single artifact: the macro you wrote in lesson 3, the docs block you wrote in lesson 4 (already resolved, with no {{ doc(...) }} left pending), and the lineage lesson 6 taught you to read with no dbt ls run at all.

Part 5 — The project's seventh commit

git status --short

What to expect.

 M models/marts/_models.yml
 M models/marts/fact_orders.sql
?? macros/calculate_revenue.sql
?? models/marts/_docs.md

Two modified files (_models.yml with the new descriptions, fact_orders.sql with the macro invocation) and two new, still-untracked files (calculate_revenue.sql, _docs.md). No complete new folder — unlike modules 4 and 5 — and no generated file (kiosko.duckdb, target/) in the list: module 1's .gitignore is still doing its job.

git add macros/calculate_revenue.sql models/marts
git commit -m "Module 7: extract calculate_revenue macro, document models and columns, generate dbt docs"

What to expect.

[master a7b8c9d] Module 7: extract calculate_revenue macro, document models and columns, generate dbt docs
 4 files changed, 46 insertions(+), 6 deletions(-)
 create mode 100644 macros/calculate_revenue.sql
 create mode 100644 models/marts/_docs.md

(The commit's short identifier, a7b8c9d in this example, is going to be different on your machine — as you already saw in earlier modules, it's a hash generated from the exact content and the moment of the commit.) Confirm the complete history and a clean working tree:

git log --oneline
git status

What to expect.

a7b8c9d (HEAD -> master) Module 7: extract calculate_revenue macro, document models and columns, generate dbt docs
f6a7b8c Module 6: make fact_orders incremental with delete+insert, prove idempotency
e5f6a7b Module 5: snapshot dim_product with SCD type 2 over the P002 change
d4e5f6a Module 4: add data tests to fact_orders, a custom generic test and a singular test
c3d4e5f Module 3: rebuild dim_store, dim_date and fact_orders with ref()
a1b2c3d Module 2: declare Kiosko sources and build the staging layer
af0b710 First dbt project: kiosko_analytics scaffolding

On branch master
nothing to commit, working tree clean

Seven commits, each documenting a real, verifiable milestone: the project's scaffolding, the staging layer, the star schema, the test suite, the automated SCD type 2, the conversion to incremental, and now the extraction of a reusable macro alongside the project's complete documentation.

Diagram: the mini-project's complete flow

flowchart TD
    A["state at the end of module 6\n7 models, 1 snapshot, 15 data tests, 6 commits"] --> B["macros/calculate_revenue.sql (L2-L3)"]
    B --> C["fact_orders.sql uses the macro\n+ full-refresh -> 40 rows, 106.15"]
    C --> D["_models.yml + _docs.md\ndescriptions and doc blocks (L4)"]
    D --> E["dbt build --vars -> PASS=23"]
    E --> F["dbt docs generate --vars\n-> manifest.json + catalog.json"]
    F --> G["inspect manifest.json:\nmacro, doc block, lineage (L6)"]
    G --> H["git add + commit -> seventh commit"]

Common mistakes

Running dbt docs generate with a run_date different from the one used in dbt build. What happens: someone runs dbt build --vars '{"run_date": "2026-08-09"}' and then dbt docs generate --vars '{"run_date": "2026-08-05"}' — a different date — and wonders why the catalog looks "outdated." Why it happens: both commands accept --vars independently, and it's easy not to notice that the specific date doesn't matter for this mini-project's purpose — all that matters is fact_orders already existing as a table, so is_incremental() is True and the var("run_date") block evaluates with no error. How to spot it: in this particular mini-project, any valid date works equally well for dbt docs generate, because the catalog introspects the table as it is right now, regardless of which partition the last run processed. How to fix it: the two dates don't need to match for this mini-project to work correctly — what matters is passing some valid date to any command that compiles fact_orders, lesson 7's central lesson.

Forgetting models/marts/_docs.md in the git add, and losing the docs block. What happens: someone runs git add macros/calculate_revenue.sql models/marts/_models.yml models/marts/fact_orders.sql — naming each specific file instead of the complete folder — and forgets _docs.md, which would end up untracked. Why it happens: _docs.md is a new file, in a different format (.md, not .sql or .yml) from the rest of this module's changes, and it's easy not to think of it as part of the same commit. How to spot it: git status after the commit would still show models/marts/_docs.md as ??, untracked. How to fix it: as this mini-project does, add the complete folder (git add models/marts, not a list of specific .sql/.yml files) so you don't lose any configuration or documentation file that goes with the SQL — the same habit module 3 already insisted on in its own mini-project.

Thinking Part 4 of this mini-project needs dbt docs serve running. What happens: someone tries to run Part 4's Python script while dbt docs serve is active in another terminal, expecting it to be a requirement. Why it happens: both — the Python script and dbt docs serve — read manifest.json, so it seems reasonable that one depends on the other. How to spot it: Part 4's script works exactly the same with or without dbt docs serve running — it reads the manifest.json file directly from disk, with no need for any active HTTP server. How to fix it: dbt docs serve is a way of visualizing the manifest in a browser; Part 4's Python script is a way of reading it directly — they're two independent consumers of the same file, neither depends on the other to work.

Exercises

Exercise 1 — Reproduce the complete mini-project from module 6's state. If you have access to a copy of the project as it was left at the end of module 6 (before this module's lesson 2), apply lessons 2 through 4's four changes without looking at the material, and reproduce this mini-project's five parts. Confirm you land at exactly PASS=23 in the final dbt build.

See solution

If you followed lessons 2 through 7's sequence, the result should be exactly the same: dbt build --vars '{"run_date": "..."}' ends at PASS=23 WARN=0 ERROR=0 SKIP=0 NO-OP=0 REUSED=0 TOTAL=23, with fact_orders showing 40 rows and 106.15 revenue, and dbt docs generate --vars '{"run_date": "..."}' producing manifest.json and catalog.json with no error at all. If the number doesn't match, first check that fact_orders.sql invokes the macro with the correct alias prefix ('o.quantity', not 'quantity'), and that you ran --full-refresh at least once after adding the macro — lesson 3 explained why a normal incremental run wasn't enough.

Exercise 2 — Simulate what would happen if a teammate cloned the repository today. With the seventh commit already made, imagine someone else clones kiosko_analytics/ for the first time. List, in order, the commands that person would need to run to reach the same state you're in right now — from git clone to dbt docs serve.

See solution
  1. git clone <url> — brings the complete source code, the seven commits, every .sql, .yml, and .md file.
  2. Create their own profiles.yml (it's not in the repository, per module 1's .gitignore) pointing at a local kiosko.duckdb.
  3. pip install dbt-core dbt-duckdb (or activate a virtual environment with both already installed).
  4. dbt build --vars '{"run_date": "..."}' — rebuilds the complete warehouse from scratch: sources, staging, the snapshot (though, as module 5 already explained, with a single dbt snapshot run over products's current state, without the two-version history you have in your own kiosko.duckdb).
  5. dbt docs generate --vars '{"run_date": "..."}' followed by dbt docs serve — to explore the complete documentation, including every description: and revenue's docs block, without having written a single line of those files themselves.

This sequence is, precisely, the whole reason for everything you versioned across the seven commits: anyone with the repository and these five steps reaches the same functional, documented, tested project, with no dependency on you explaining anything to them in person.

Exercise 3 — Argue why this mini-project is the right close before module 8's capstone. In 2-3 sentences, explain what guarantees you get from having the complete project — macro, documentation, verified lineage, seven commits — before module 8 ports the remaining marts from data-modeling-for-analytics-guide.

See solution

Module 8 is going to add four new marts (dim_category, dim_order_flags, fact_sessions, fact_store_activity) and the final report (mart_daily_sales_obt), reusing exactly the same pattern modules 1 through 7 already validated — ref(), materializations, tests, documentation — not a new pattern to learn from scratch. Having calculate_revenue already extracted and documented means any new model that also needs to calculate revenue can reuse the same macro, instead of repeating the formula a third time; and having the habit of documenting every model and column, already practiced in this module over three marts, makes documenting five more marts in module 8 a repetition of the same process, not a new effort to invent on the fly.

Summary and next step

In this mini-project you confirmed that Kiosko's complete project, with the calculate_revenue macro and the complete documentation from lessons 2 through 4, still produces exactly the same result as always: dbt build --vars ends at PASS=23, fact_orders has 40 rows and 106.15 revenue. You generated manifest.json and catalog.json with dbt docs generate, and inspected, in a single script, this module's three new pieces inside the same artifact: the macro fact_orders invokes, the revenue column's already-resolved docs block, and the model's complete lineage. You closed the module with the project's seventh commit.

With this you close module 7. You now have a kiosko_analytics/ project with seven models, one snapshot, fifteen data_tests, one reusable macro, complete model- and column-level documentation, and seven commits in its history, each documenting a real, verifiable milestone.

Where you go next. Module 8 is this guide's capstone: you're going to port the remaining marts data-modeling-for-analytics-guide already designed and verified — dim_category, dim_order_flags, fact_sessions, fact_store_activity, mart_daily_sales_obt — reusing exactly the same pattern modules 1 through 7 already taught you, with no new dbt concept to learn. The module closes with dbt build running end to end over the complete project — seeds, sources, snapshot, every model, every test — and a final query over mart_daily_sales_obt, the same sales report data-modeling-for-analytics-guide produced by hand, now reproducible by anyone with a single command.

Resources

← PreviousWhen a dbt project needs a schedulerNext →Module 8 — Project: Kiosko's first dbt warehouse