Reading the lineage graph

Description

Since module 3, dbt ls --select +fact_orders has been your tool for seeing what a model depends on — a command that asks dbt, on the spot, what the dependency tree is. This lesson learns a different way of answering the same question: reading the lineage directly from manifest.json, the artifact dbt docs generate already produced in lesson 5. The difference isn't cosmetic. A command like dbt ls runs once and disappears; an artifact like manifest.json stays on disk, can be versioned, compared between two runs, or feed an external tool that doesn't even have dbt installed — all it needs is a JSON file and the structure this lesson teaches you to read.

Connection to the module. Lesson 5 generated the artifacts; this lesson reads them for a specific purpose — rebuilding, with no dbt command run at all, the same dependency tree you already know from module 3. And it closes the module's analogy's circle: lineage as a subway map, not as a hand-drawn diagram — the colored lines are already in manifest.json, all you have to do is follow them.

Two directions of the same graph: depends_on and child_map

manifest.json stores dependency relationships in two different directions, and each answers a different question:

• depends_on.nodes, inside each node, answers "what does this model depend on?" — looking backward, toward its ancestors. You already saw fact_orders["depends_on"]["nodes"] in lesson 5: stg_orders, dim_store, dim_date.
• child_map, at the complete manifest's level (not inside each node), answers "what depends on this model?" — looking forward, toward its descendants. child_map["model.kiosko_analytics.dim_store"] tells you which models and tests depend on dim_store.

The same graph's two directions, stored separately so you don't have to walk the whole manifest every time you need one of the two questions.

Worked example: fact_orders's complete tree

Rebuild fact_orders's dependency tree with a recursive script, with no dbt ls run at all:

import json

manifest = json.load(open("target/manifest.json"))
nodes = manifest["nodes"]
sources = manifest["sources"]

def label(unique_id):
    if unique_id in nodes:
        return nodes[unique_id]["name"]
    if unique_id in sources:
        source = sources[unique_id]
        return f"source:{source['source_name']}.{source['name']}"
    return unique_id

def print_tree(unique_id, indent=0):
    print("  " * indent + "- " + label(unique_id))
    node = nodes.get(unique_id)
    if not node:
        return
    for dependency in node.get("depends_on", {}).get("nodes", []):
        print_tree(dependency, indent + 1)

print_tree("model.kiosko_analytics.fact_orders")

What to expect.

- fact_orders
  - stg_orders
    - source:kiosko_raw.orders
  - dim_store
    - stg_stores
      - source:kiosko_raw.stores
  - dim_date

Compare this against dbt ls --select +fact_orders, exactly as you ran it in module 3: the dependency tree is the same, but the count doesn't match number for number, and it's worth distinguishing why. This script follows only models' and sources' depends_on.nodes — never tests — so it counts seven nodes in total (fact_orders, stg_orders, dim_store, stg_stores, dim_date, and the two sources), rebuilt entirely from a JSON file, with no additional dbt command. dbt ls --select +fact_orders, by contrast, reported eleven resources in module 3 — the same command includes, through indirect selection, the four staging data_tests that depend on those same models. This script's seven nodes are the pure data graph; dbt ls's eleven are that same graph plus the tests that validate it — this lesson's Common mistakes section comes back to this same distinction in more detail. dim_date has no branch below it — it doesn't depend on any ref() or source(), exactly as you already know from module 3: it's generated with a pure recursive CTE, with no external dependency at all.

The new piece: macros inside the tree

dbt ls never showed what macros a model depends on — only what other models and sources. manifest.json does store it, in the same depends_on field you already used to rebuild the tree above:

fact_orders = nodes["model.kiosko_analytics.fact_orders"]
print("Macros fact_orders invokes:")
for macro_id in fact_orders["depends_on"]["macros"]:
    print(" -", macro_id)

What to expect.

Macros fact_orders invokes:
 - macro.kiosko_analytics.calculate_revenue
 - macro.dbt.is_incremental

Two macros: calculate_revenue, the one you wrote in lesson 3, with the kiosko_analytics prefix because it's specific to the project; and is_incremental, with the dbt prefix because it ships with dbt-core, the same macro all of module 6 used without you ever having to write it. This is the evidence, inside the artifact itself, of why extracting calculate_revenue in lesson 3 was a real, traceable change: before that lesson, this same script would have only shown is_incremental in the list.

Looking forward: what depends on dim_store

Invert the question with child_map, at the complete manifest's level:

child_map = manifest["child_map"]
print("What depends on dim_store:")
for child_id in child_map["model.kiosko_analytics.dim_store"]:
    print(" -", child_id)

What to expect (each resource's complete unique identifier, not resolved to a short name):

What depends on dim_store:
 - model.kiosko_analytics.fact_orders
 - test.kiosko_analytics.relationships_fact_orders_store_id__store_id__ref_dim_store_.c14f2be544

Two descendants: fact_orders (the model you already knew) and module 4's relationships test — the one that validates every store_id in fact_orders exists in dim_store. Notice each identifier's prefix: model.kiosko_analytics.... versus test.kiosko_analytics...., with an additional hexadecimal suffix on the second one (c14f2be544 in this example — it's going to be different on your machine, it's a hash calculated from the test's exact configuration). This confirms something you already knew conceptually but had never seen made explicit in an artifact: a data_test is, for the dependency graph's purposes, just another node, with its own unique identifier and its own entry in depends_on — it's not a property attached to dim_store, it's an independent resource that depends on it. If you wanted the short name instead of the complete identifier, the same label() function from the earlier example resolves it: for both models and tests, it returns nodes[unique_id]["name"].

Lineage as a diagram, generated from the same artifact

The tree you rebuilt above can be expressed as a diagram, straight from the same information:

flowchart LR
    SO["source:kiosko_raw.orders"] --> STO[stg_orders]
    SS["source:kiosko_raw.stores"] --> STS[stg_stores]
    STS --> DS[dim_store]
    STO --> FO[fact_orders]
    DS --> FO
    DD[dim_date] --> FO
    FO -.calculate_revenue.-> FO

The dotted arrow in fact_orders -.calculate_revenue.-> fact_orders represents something different from the others: it's not a dependency on another model, it's a dependency on a macro invoked inside the file itself — that's why it points to the same node, and that's why dbt docs serve doesn't draw it in its visual graph (which only shows models, sources, and tests, not macros). This is exactly the advantage of reading the artifact instead of only looking at the diagram: manifest.json stores information — like macro dependencies — that not even the browsable documentation site chooses to show visually.

Why this is "lineage as an artifact," not as a hand-drawn diagram

Go back, for a moment, to the module introduction's analogy: dbt docs's lineage is the subway map, not a blueprint someone draws by memorizing every tunnel. Every time you add a new model with a new ref() or source(), that change gets automatically reflected in manifest.json the next time you run dbt docs generate — nobody has to update any diagram by hand, because the diagram never existed as an object separate from the code: it's a projection of the code, rebuilt from scratch on every run. If module 8 adds dim_category, fact_sessions, and the remaining marts tomorrow, this same lesson's script — with no change at all — would rebuild a bigger tree, automatically, just by pointing at a regenerated manifest.json.

Common mistakes

Looking for macro dependencies inside child_map, instead of depends_on. What happens: someone, after seeing child_map answers "what depends on this node," tries using it to find "which models use the calculate_revenue macro." Why it happens: child_map does include models and tests as descendants of other models, so it seems reasonable to expect it to also work for macros. How to spot it: manifest["child_map"].get("macro.kiosko_analytics.calculate_revenue") returns an empty list or None — child_map only tracks relationships between executable nodes (models, sources, tests, snapshots), not between a macro and whoever invokes it. How to fix it: to find which models use a specific macro, walk the complete manifest["nodes"], looking for that macro inside each node's depends_on.macros field — the same pattern you already used to read fact_orders's macros, applied in reverse.

Assuming depends_on.nodes's order reflects the DAG's execution order. What happens: someone reads fact_orders's depends_on.nodes (stg_orders, dim_store, dim_date) and assumes that's the order dbt builds those three models in. Why it happens: the list looks ordered, and it's easy to confuse "order they appear in the file" with "execution order." How to spot it: check dbt run's real output over the complete project, since module 3 — dim_date and the staging views run in parallel, with no fixed order between them, because neither depends on the other. How to fix it: depends_on.nodes is a set of dependencies, not a sequence — the real execution order gets decided by dbt's engine at run time, based on how many threads you have configured (profiles.yml, module 1) and how soon each dependency frees up, never based on the order they appear in the manifest's list.

Comparing this script's node count against dbt ls --select +fact_orders's, and expecting them to match exactly. What happens: someone runs both — this lesson's script and dbt ls --select +fact_orders — and is surprised if the numbers don't match on some project with more declared tests than this lesson's. Why it happens: dbt ls with the + operator includes, by default behavior, any test whose dependencies are all already selected — a rule called indirect selection — while this lesson's script only follows models' and sources' depends_on.nodes, with no tests. How to spot it: if a project has tests declared on fact_orders or on any of its ancestors, dbt ls --select +fact_orders may include them in its list, while this script never shows them — because a test is a model's descendant, not an ancestor. How to fix it: both are correct, they answer different questions: dbt ls --select +fact_orders answers "what resources do I need to build or run to have fact_orders ready and tested"; this lesson's script answers, precisely, "what other models and sources does fact_orders's SELECT depend on" — the pure data graph, without mixing in the tests that validate it.

Exercises

Exercise 1 — Rebuild dim_store's tree. Using the worked example's print_tree function, print dim_store's dependency tree (not fact_orders's). How many levels does it have?

print_tree("model.kiosko_analytics.dim_store")

- dim_store
  - stg_stores
    - source:kiosko_raw.stores

Exercise 2 — Find every model that invokes is_incremental. Walk manifest["nodes"], and for each node of type model, check whether is_incremental shows up in its depends_on.macros. How many models use it today?

for node_id, node in nodes.items():
    if node["resource_type"] != "model":
        continue
    macro_ids = node.get("depends_on", {}).get("macros", [])
    if any("is_incremental" in m for m in macro_ids):
        print(node["name"])

Exercise 3 — Argue why manifest.json is more useful than dbt ls for an external tool. In 2-3 sentences, explain why a data governance system, or a corporate catalog that consolidates lineage across several dbt projects at once, would prefer reading JSON artifacts instead of repeatedly running dbt ls.

Summary and next step

In this lesson you read fact_orders's lineage directly from manifest.json, with no dbt ls run at all: you rebuilt its complete dependency tree with a recursive script over depends_on.nodes, found its two macro dependencies with depends_on.macros — including calculate_revenue, the evidence inside the artifact itself of the change you made in lesson 3 — and inverted the question with child_map to see what depends on dim_store. You confirmed the same tree you already knew from module 3 lives, complete, inside a versionable JSON file.

Before moving on you should be able to: explain the difference between depends_on.nodes (looking backward) and child_map (looking forward); and name why a data_test shows up as an independent node in the graph, not as a property of the model it tests.

Lesson 7 changes the topic for the last time in this module: you're going to name, with evidence you've already lived through in modules 6 and 7, the exact moment running all of this by hand — dbt build, dbt docs generate, remembering --vars every time — stops being enough, and where this guide points when that happens.

Resources

• dbt Developer Hub — "Manifest," manifest.json's complete schema, including depends_on, parent_map, and child_map. docs.getdbt.com/reference/artifacts/manifest-json. In English.
• dbt Developer Hub — "Node selection syntax," the section on indirect selection, the rule that explains why dbt ls --select +fact_orders can include tests this lesson's script doesn't show. docs.getdbt.com/reference/node-selection/syntax. In English.
• data-reliability-and-governance-guide, the ecosystem's sibling guide that extends a single project's lineage into a governance catalog across several systems and tools.