Module 7 — Macros, documentation, and lineage

Description

Since module 3, fact_orders.sql has calculated each order line's revenue with o.quantity * o.unit_price as revenue — a two-column expression, written directly inside the SELECT. It works perfectly well that way, and nothing in modules 3 through 6 questioned it. But imagine Kiosko grew: a new model summarizing revenue by store, another comparing projected revenue against actual revenue, a third one for the finance team that needs the same calculation with an extra discount column. Every one of those future models would have to repeat quantity * unit_price, or some variation of that formula, written by hand, over and over. And the day the business decides revenue should also subtract returns, or apply a tax, someone would have to find every copy of that expression, in every file, and fix them all at once — with no guarantee of not missing one.

This module solves that problem with the whole guide's first real piece of "reusable code": dbt macros, Jinja templates written once and invoked from any model. You're going to write calculate_revenue, the macro that replaces fact_orders.sql's inline expression, verifying with real, executed evidence that the result doesn't change by a single cent (40 rows, 106.15 revenue, the exact same number since module 3). Then you're going to document every model and every column in the project with description:, you're going to automatically generate the project's complete documentation with dbt docs generate — literally inspecting the two JSON files it produces, manifest.json and catalog.json — and you're going to read the project's lineage directly from those artifacts, with no diagram drawn by hand. The module closes by naming, precisely, the exact moment running dbt build by hand from your terminal stops being enough — and where this guide points when that happens.

Connection to the previous module. Module 6 turned fact_orders into an incremental model and proved, with repeated row counts, that running the same command twice doesn't duplicate anything — the idempotency property. That module also left a clue, still unresolved: Exercise 3 of its mini-project hinted that "module 7 is going to introduce a reusable macro that's likely going to replace o.quantity * o.unit_price." This module delivers exactly on that promise, and uses the operational tools module 6 already gave you — --full-refresh, reading a cascading ERROR — to apply that change with the same discipline you already practiced.

Three analogies for three ideas in this module

A macro is a formula you write once and use in ten documents, instead of copying it by hand into each one. Think of a tax calculator an accountant programs once in a spreadsheet — with one cell for revenue and another for the rate — and then drags across hundreds of different rows. If the tax rate changes tomorrow, they fix the formula in a single place, and the hundreds of rows that use it update automatically. Copying the formula by hand, row by row, would work just as well on day one — until something changed, and you'd have to find every copy, one by one, with no guarantee of not leaving an old one behind.

Documenting a model is putting a label on every jar in the pantry, not trusting everyone to remember what's inside. An unlabeled jar forces you to open it and taste it to know what it contains — that works while you're the only person cooking in that kitchen. As soon as someone else walks into that kitchen, every unlabeled jar is a question that has to be asked out loud. description: in _models.yml is, precisely, that label: once written, any teammate who opens dbt docs serve knows what dim_date.is_weekend contains without having to ask you or open the .sql file.

The lineage dbt docs generate produces is the subway map, not a diagram someone draws by hand that goes stale the moment something changes. A subway map doesn't memorize each individual tunnel — it follows the colored lines that already exist, trusting they reflect the real tracks. dbt's lineage works the same way: it's not a diagram someone draws once on a whiteboard and that goes obsolete the moment you add a new model — it's an artifact (manifest.json) dbt rebuilds, complete and up to date, every time you run dbt docs generate, from the ref() and source() lines you already wrote in your models. You never have to "update the diagram" by hand — the diagram is the code.

What you're going to build in this module

You're going to extract fact_orders.sql's revenue calculation into a reusable macro, document the complete project, and generate its documentation and lineage as real artifacts:

1. A practice macro (discounted_price) to learn the {% macro %} / {% endmacro %} syntax without touching Kiosko's project yet.
2. macros/calculate_revenue.sql — the real macro, applied to fact_orders.sql in place of the inline expression, with --full-refresh to propagate the change to all 40 rows.
3. description: on every model and column in models/marts/_models.yml, including a reusable docs block ({% docs %} / {% enddocs %}) for the revenue column.
4. dbt docs generate — run, with a literal inspection of manifest.json and catalog.json.
5. A script that rebuilds fact_orders's dependency tree by reading manifest.json directly — lineage as an artifact, not as a command.
6. The explicit bridge toward airflow-and-declarative-orchestration-guide, with evidence you've already lived through of why running everything by hand, from memory, stops being enough.

flowchart TD
    A["fact_orders.sql\ninline revenue (modules 3-6)"] --> B["M7 L2: macro syntax\n(practice, discounted_price)"]
    B --> C["M7 L3: calculate_revenue\n+ full-refresh -> 40 rows, 106.15"]
    C --> D["M7 L4: description: on\nmodels, columns, and doc blocks"]
    D --> E["M7 L5: dbt docs generate\n-> manifest.json + catalog.json"]
    E --> F["M7 L6: read the lineage\nfrom the artifact"]
    F --> G["M7 L7: the limit of\nrunning everything by hand"]
    G --> H["M7 L8: complete project\nseventh commit"]

This module's 8 lessons

#	Lesson	What it solves
1	Module introduction	This map.
2	Writing your first macro	{% macro %}/{% endmacro %} syntax, invocation with {{ }}, over a practice macro.
3	A reusable revenue macro for Kiosko	calculate_revenue, really applied to fact_orders.sql, with --full-refresh.
4	Documenting models and columns	description: on models and columns, plus reusable docs blocks.
5	Generating docs with dbt docs generate	The command run, manifest.json and catalog.json as real artifacts.
6	Reading the lineage graph	fact_orders's dependency tree, rebuilt directly from manifest.json.
7	When a dbt project needs a scheduler	The explicit bridge toward airflow-and-declarative-orchestration-guide.
8	Mini-project: Kiosko's documented project	The whole module, end to end, with the project's seventh commit.

This module's boundary

This module ends where any manual execution ends: you trigger dbt build from your own terminal, and dbt docs generate produces artifacts that live on your disk, in your machine's target/ folder. What it does not cover is who schedules that run to happen on its own every morning, who automatically retries if something fails halfway through, or how that documentation gets published on a server the rest of the team can visit without having run the project themselves. That's a scheduler's job — dbt Cloud, or Airflow running dbt build as a task in a larger DAG, with retries, sensors, and separate development and production environments managed by the scheduler, not by you by hand — and it's exactly the topic of airflow-and-declarative-orchestration-guide, the guide that follows this one in the ecosystem. This module's lesson 7 is the explicit bridge toward it: it names, with evidence you've already lived through in modules 6 and 7, the precise moment running everything from memory stops being enough.

Why calculate_revenue, and not some other kind of macro

Every earlier module in this guide chose, deliberately, to solve a real, visible problem in Kiosko's project, never a disconnected toy example. This module follows that same discipline: calculate_revenue isn't a made-up example to demonstrate macro syntax — it's the extraction of an expression that already exists, repeated with no reuse mechanism at all, since module 3. Choosing it as this module's central example has a concrete advantage: you already know, from memory, what the correct result is (40 rows, 106.15 total revenue) before writing a single line of Jinja — so any deviation in the result, after the change, is an immediate signal that something went wrong in the macro, not a new number you'd have to calculate from scratch to confirm is correct.

Common mistakes

Thinking a macro is just a way to "save typing". What happens: someone sees calculate_revenue('o.quantity', 'o.unit_price') — more characters than o.quantity * o.unit_price — and concludes the macro adds no real value, because the resulting file isn't even shorter. Why it happens: a macro's payoff isn't visible in a single file, with a single invocation — it's invisible until there's a second invocation, in a different model, and it becomes decisive the day the formula needs to change in both places at once. How to spot it: count how many files in your project use the same repeated math expression — with only fact_orders.sql, calculate_revenue's payoff is, in fact, minimal. How to fix it: judge a macro's value by what it avoids the day something changes, not by how many characters it saves today — this lesson's Description already previewed the real scenario: a new finance model that needs the same calculation, tomorrow, without copying the formula by hand.

Confusing documenting a model with testing it. What happens: someone finishes this module and assumes that, because fact_orders now has a complete description: on every column, the model is somehow "more verified" than before. Why it happens: documentation and tests show up together in _models.yml, in the same YAML file, and it's easy to mix the two concepts. How to spot it: ask yourself whether description: ever makes dbt test fail — it never does; an incorrect or outdated description doesn't break any run, or generate any WARN. How to fix it: this module's documentation answers "what does this column mean?" for a human reading dbt docs serve; module 4's data_tests: answer "does the data still follow this rule?" for an automated run — they're two different layers of the same project, and neither replaces the other.

Assuming dbt docs generate is only for large projects, with many models. What happens: someone, seeing that Kiosko has barely seven models, decides generating formal documentation is premature — "I already know from memory what every model does, I don't need a website to remind me." Why it happens: with a project small enough to fit in a single person's head, generated documentation's value isn't obvious yet. How to spot it: ask yourself what would happen if tomorrow a teammate, who never saw this project, had to understand what dim_date.is_weekend is without asking you directly. How to fix it: the habit of documenting — just like module 4's habit of testing — gets built while the project is small and manageable, not once it's already grown large enough that documenting it retroactively becomes a whole project in itself.

Exercises

Exercise 1 — Locate the expression this module is going to extract. Before starting lesson 2, open models/marts/fact_orders.sql exactly as it was left at the end of module 6, and copy the exact line that calculates revenue. You're going to compare that line, word for word, against the version lesson 3 is going to produce.

See solution

The exact line is o.quantity * o.unit_price as revenue,, inside fact_orders.sql's select block, unchanged since data-modeling-for-analytics-guide originally designed it and this guide's module 3 ported it to dbt. Lesson 3 is going to replace only the o.quantity * o.unit_price expression with a macro invocation — the as revenue alias and the rest of the file stay exactly the same.

Exercise 2 — Predict which files this module is going to touch. Based on the 8-lesson table, predict: which project folders (macros/, models/marts/, snapshots/, tests/) is this module going to modify or create, and which stay exactly the same?

See solution

This module creates macros/calculate_revenue.sql (new), modifies models/marts/fact_orders.sql (the revenue line) and models/marts/_models.yml (new descriptions), and adds models/marts/_docs.md (new, with the reusable docs block). snapshots/ and tests/ don't change at all — no lesson in this module touches dim_product's snapshot or the fifteen-data_tests suite module 4 left in place; both keep working exactly the same, verified again in lesson 8's mini-project.

Exercise 3 — Explain, in your own words, the difference between what a macro solves and what ref() solves. In 2-3 sentences, using what you already know from module 3 (ref(), the DAG) and this module's Description (macros, reusing logic), explain why both mechanisms are ways of "not repeating things," but solve different problems.

See solution

ref() avoids repeating data: instead of every model recalculating dim_store from scratch, ref('dim_store') reuses the table another model already built, and gives dbt the information it needs to order runs correctly. A macro avoids repeating logic: instead of writing the same SQL expression (quantity * unit_price) in every file that needs it, calculate_revenue() centralizes that formula in a single place, and any model that invokes it gets the same calculation, without copying a single line. They're complementary, not alternatives: fact_orders.sql, by the end of this module, is going to use both mechanisms at once — ref() for its three model dependencies, and calculate_revenue() for its revenue calculation.

Summary and next step

This module is going to extract fact_orders.sql's revenue calculation into a reusable macro (calculate_revenue), verified against the same exact result as always (40 rows, 106.15 revenue). You're going to document every model and column in the project with description: and reusable docs blocks, generate the complete documentation with dbt docs generate — literally inspecting the manifest.json and catalog.json artifacts it produces — and read the project's lineage directly from those files, with no diagram drawn by hand. The module closes by naming the explicit bridge toward real orchestration.

Before moving on to lesson 2 you should be able to: explain, in your own words, the difference between what ref() solves and what a macro solves; and name the exact expression this module is going to extract from fact_orders.sql.

Lesson 2 starts at the beginning: a Jinja macro's minimal syntax, over a practice example that doesn't touch any file in Kiosko's project yet.

Resources

• dbt Developer Hub — "Jinja and macros," this whole module's central syntax reference. docs.getdbt.com/docs/build/jinja-macros. In English.
• dbt Developer Hub — "About documentation," an overview of what dbt docs generate produces and how description: relates to docs blocks. docs.getdbt.com/docs/collaborate/documentation. In English.
• airflow-and-declarative-orchestration-guide, the next guide in the ecosystem, which this module's lesson 7 explicitly points toward.