Module 2: Sources And Staging Models

Declaring Kiosko's raw data as sources

Next →

Description

Before declaring a single source, there are two housekeeping tasks to do: delete the models/example/ folder module 1 left pending (it already served its purpose; keeping it around only adds noise), and decide where, physically, Kiosko's raw files are going to live inside your project — a decision that, on purpose, isn't the seeds/ folder you already reserved in dbt_project.yml since module 1.

This lesson does both things and ends with this whole guide's first sources.yml file: the declaration — still without the reading mechanism, which arrives in lesson 4 — of the four raw files this module is going to turn into staging models.

Connection to the module. Lesson 1 gave you the complete map and the receiving-dock analogy. This lesson builds that dock's first concrete piece: the register with the names of what's going to come in. You still won't be able to read a single row of real data — that starts in lesson 4 — but by the end of this lesson dbt is already going to recognize the four sources as a valid part of your project.

Step 0: clean up what module 1 left pending

From kiosko_analytics/'s root, delete the folder that held module 1's trivial model:

rm -rf models/example

Module 1's lesson 8 already anticipated this step, so it shouldn't surprise you: my_first_dbt_model served its one purpose — demonstrating that dbt's mechanism works end to end — and contributes nothing from here on. If you run dbt run right now, you're going to see Found 0 models — a completely empty models/ folder, waiting for the real files this module is going to create.

Where raw files live: raw_data/, not seeds/

dbt_project.yml already declares, since module 1, a folder reserved for CSV files: seed-paths: ["seeds"]. It might seem like the obvious place for Kiosko's raw files — but it isn't, and it's worth understanding why before creating a single folder.

seeds/ has a specific, different meaning inside dbt: any CSV you put there, dbt loads into the warehouse with the dbt seed command — it literally reads the file and generates the INSERTs needed to create a new table with that data. It's the mechanism meant for small, static catalogs you yourself maintain as part of the project (for example, a mapping table of country codes to names). Kiosko's files are a different animal entirely: they're raw data that already exists outside dbt, generated by an external system (in this case, data-engineering-foundations-guide), and this whole module's goal is to read them without loading them, straight from the file. If you put orders_2026-08-03.csv inside seeds/, accidentally running dbt seed would create a completely redundant orders_2026_08_03 table — exactly what this module exists to avoid.

That's why this module introduces a new folder, outside any *-paths dbt reserves, so it's clear — both to dbt and to anyone reading the project — that these files are external data, not seeds or models:

mkdir -p raw_data/kiosko

Worked example: Kiosko's four raw files

Inside raw_data/kiosko/, you're going to recreate exactly the files you already generated in data-engineering-foundations-guide — without changing a single row — plus a new column in products_v1.csv this guide needs starting in module 5.

orders: seven files, one per day

The full week you already worked in foundations, from 2026-08-03 to 2026-08-09, with no changes:

-- raw_data/kiosko/orders_2026-08-03.csv
order_id,store_id,product_id,quantity,unit_price,order_ts
ORD-1001,S01,P001,3,0.55,2026-08-03T08:14:00
ORD-1002,S01,P002,1,1.20,2026-08-03T08:20:00
ORD-1003,S02,P003,2,0.75,2026-08-03T08:31:00
ORD-1004,S01,P004,1,4.50,2026-08-03T09:02:00
ORD-1005,S03,P001,5,0.55,2026-08-03T09:15:00
ORD-1006,S02,P002,2,1.20,2026-08-03T09:47:00
ORD-1007,S03,P003,1,0.75,2026-08-03T10:05:00
ORD-1008,S01,P001,2,0.55,2026-08-03T10:22:00

The other six files follow the same exact header (order_id,store_id,product_id,quantity,unit_price,order_ts), with the same content you already generated in foundations:

-- raw_data/kiosko/orders_2026-08-04.csv
order_id,store_id,product_id,quantity,unit_price,order_ts
ORD-2001,S01,P002,1,1.20,2026-08-04T08:05:00
ORD-2002,S02,P001,4,0.55,2026-08-04T08:40:00
ORD-2003,S03,P004,1,4.50,2026-08-04T09:12:00
ORD-2004,S01,P003,3,0.75,2026-08-04T09:50:00
ORD-2005,S02,P002,2,1.20,2026-08-04T10:15:00
ORD-2006,S03,P001,6,0.55,2026-08-04T10:33:00

-- raw_data/kiosko/orders_2026-08-05.csv
order_id,store_id,product_id,quantity,unit_price,order_ts
ORD-3001,S02,P004,2,4.50,2026-08-05T08:10:00
ORD-3002,S01,P001,1,0.55,2026-08-05T08:22:00

-- raw_data/kiosko/orders_2026-08-06.csv
order_id,store_id,product_id,quantity,unit_price,order_ts
ORD-4001,S01,P001,4,0.55,2026-08-06T08:10:00
ORD-4002,S02,P003,2,0.75,2026-08-06T08:45:00
ORD-4003,S03,P002,1,1.20,2026-08-06T09:20:00
ORD-4004,S01,P004,1,4.50,2026-08-06T09:55:00
ORD-4005,S02,P001,3,0.55,2026-08-06T10:30:00

-- raw_data/kiosko/orders_2026-08-07.csv
order_id,store_id,product_id,quantity,unit_price,order_ts
ORD-5001,S01,P002,2,1.20,2026-08-07T08:05:00
ORD-5002,S03,P001,4,0.55,2026-08-07T08:30:00
ORD-5003,S02,P004,1,4.50,2026-08-07T08:58:00
ORD-5004,S01,P003,2,0.75,2026-08-07T09:22:00
ORD-5005,S03,P002,3,1.20,2026-08-07T09:47:00
ORD-5006,S02,P001,5,0.55,2026-08-07T10:15:00
ORD-5007,S01,P001,2,0.55,2026-08-07T10:40:00

-- raw_data/kiosko/orders_2026-08-08.csv
order_id,store_id,product_id,quantity,unit_price,order_ts
ORD-6001,S01,P001,6,0.55,2026-08-08T08:00:00
ORD-6002,S02,P002,3,1.20,2026-08-08T08:18:00
ORD-6003,S03,P001,4,0.55,2026-08-08T08:35:00
ORD-6004,S01,P004,2,4.50,2026-08-08T08:52:00
ORD-6005,S02,P003,3,0.75,2026-08-08T09:10:00
ORD-6006,S03,P002,2,1.20,2026-08-08T09:28:00
ORD-6007,S01,P003,1,0.75,2026-08-08T09:45:00
ORD-6008,S02,P001,7,0.55,2026-08-08T10:02:00
ORD-6009,S03,P004,1,4.50,2026-08-08T10:20:00

-- raw_data/kiosko/orders_2026-08-09.csv
order_id,store_id,product_id,quantity,unit_price,order_ts
ORD-7001,S01,P001,2,0.55,2026-08-09T09:15:00
ORD-7002,S02,P002,1,1.20,2026-08-09T09:40:00
ORD-7003,S03,P001,3,0.55,2026-08-09T10:05:00

Seven files, 8+6+2+5+7+9+3 = 40 rows total — the same number you already verified in foundations with parse_week.py. Note that number; in lesson 7 you're going to confirm it again, this time with dbt run.

events: the same pattern, applied to the clickstream

Foundations only showed you a sample of four events from the first day, to teach the JSON Lines format. This guide completes the entire week, with the exact same schema (event_id, event_type, session_id, event_ts) and without touching the four lines you already know:

// raw_data/kiosko/events_2026-08-03.jsonl
{"event_id": "E5001", "event_type": "page_view", "session_id": "SESS-01", "event_ts": "2026-08-03T08:00:12"}
{"event_id": "E5002", "event_type": "add_to_cart", "session_id": "SESS-01", "event_ts": "2026-08-03T08:02:45"}
{"event_id": "E5003", "event_type": "purchase", "session_id": "SESS-01", "event_ts": "2026-08-03T08:03:10"}
{"event_id": "E5004", "event_type": "page_view", "session_id": "SESS-02", "event_ts": "2026-08-03T08:05:00"}
// raw_data/kiosko/events_2026-08-04.jsonl
{"event_id": "E5005", "event_type": "page_view", "session_id": "SESS-03", "event_ts": "2026-08-04T08:10:00"}
{"event_id": "E5006", "event_type": "add_to_cart", "session_id": "SESS-03", "event_ts": "2026-08-04T08:12:30"}
{"event_id": "E5007", "event_type": "purchase", "session_id": "SESS-03", "event_ts": "2026-08-04T08:13:05"}
{"event_id": "E5008", "event_type": "page_view", "session_id": "SESS-04", "event_ts": "2026-08-04T08:20:00"}
{"event_id": "E5009", "event_type": "page_view", "session_id": "SESS-05", "event_ts": "2026-08-04T08:45:00"}
// raw_data/kiosko/events_2026-08-05.jsonl
{"event_id": "E5010", "event_type": "page_view", "session_id": "SESS-06", "event_ts": "2026-08-05T08:00:00"}
{"event_id": "E5011", "event_type": "page_view", "session_id": "SESS-07", "event_ts": "2026-08-05T08:15:00"}
{"event_id": "E5012", "event_type": "add_to_cart", "session_id": "SESS-07", "event_ts": "2026-08-05T08:16:20"}
// raw_data/kiosko/events_2026-08-06.jsonl
{"event_id": "E5013", "event_type": "page_view", "session_id": "SESS-08", "event_ts": "2026-08-06T08:05:00"}
{"event_id": "E5014", "event_type": "add_to_cart", "session_id": "SESS-08", "event_ts": "2026-08-06T08:07:15"}
{"event_id": "E5015", "event_type": "purchase", "session_id": "SESS-08", "event_ts": "2026-08-06T08:08:00"}
{"event_id": "E5016", "event_type": "page_view", "session_id": "SESS-09", "event_ts": "2026-08-06T08:30:00"}
// raw_data/kiosko/events_2026-08-07.jsonl
{"event_id": "E5017", "event_type": "page_view", "session_id": "SESS-10", "event_ts": "2026-08-07T08:00:00"}
{"event_id": "E5018", "event_type": "add_to_cart", "session_id": "SESS-10", "event_ts": "2026-08-07T08:03:10"}
{"event_id": "E5019", "event_type": "purchase", "session_id": "SESS-10", "event_ts": "2026-08-07T08:04:00"}
{"event_id": "E5020", "event_type": "page_view", "session_id": "SESS-11", "event_ts": "2026-08-07T08:20:00"}
{"event_id": "E5021", "event_type": "add_to_cart", "session_id": "SESS-11", "event_ts": "2026-08-07T08:22:00"}
{"event_id": "E5022", "event_type": "page_view", "session_id": "SESS-12", "event_ts": "2026-08-07T08:50:00"}
// raw_data/kiosko/events_2026-08-08.jsonl
{"event_id": "E5023", "event_type": "page_view", "session_id": "SESS-13", "event_ts": "2026-08-08T07:55:00"}
{"event_id": "E5024", "event_type": "add_to_cart", "session_id": "SESS-13", "event_ts": "2026-08-08T07:58:00"}
{"event_id": "E5025", "event_type": "purchase", "session_id": "SESS-13", "event_ts": "2026-08-08T07:59:10"}
{"event_id": "E5026", "event_type": "page_view", "session_id": "SESS-14", "event_ts": "2026-08-08T08:10:00"}
{"event_id": "E5027", "event_type": "add_to_cart", "session_id": "SESS-14", "event_ts": "2026-08-08T08:12:45"}
{"event_id": "E5028", "event_type": "purchase", "session_id": "SESS-14", "event_ts": "2026-08-08T08:13:30"}
{"event_id": "E5029", "event_type": "page_view", "session_id": "SESS-15", "event_ts": "2026-08-08T08:40:00"}
// raw_data/kiosko/events_2026-08-09.jsonl
{"event_id": "E5030", "event_type": "page_view", "session_id": "SESS-16", "event_ts": "2026-08-09T09:00:00"}
{"event_id": "E5031", "event_type": "page_view", "session_id": "SESS-17", "event_ts": "2026-08-09T09:20:00"}
{"event_id": "E5032", "event_type": "add_to_cart", "session_id": "SESS-17", "event_ts": "2026-08-09T09:22:00"}

4+5+3+4+6+7+3 = 32 events total (17 page_view, 9 add_to_cart, 6 purchase — you're going to verify this exact breakdown in lesson 7).

stores and products: catalogs, one single file each

-- raw_data/kiosko/stores.csv
store_id,store_name,city
S01,Kiosko Centro,Bogota
S02,Kiosko Norte,Lima
S03,Kiosko Sur,Santiago

-- raw_data/kiosko/products_v1.csv
product_id,product_name,category,unit_cost,product_updated_at
P001,Bottled Water 600ml,beverages,0.40,2026-08-01
P002,Energy Bar,snacks,0.60,2026-08-01
P003,Instant Coffee Sachet,beverages,0.35,2026-08-01
P004,Phone Charger Cable,electronics,2.10,2026-08-01

Notice products_v1.csv's new column: product_updated_at. It didn't exist in foundations, because foundations never needed to historize anything. This guide's module 5 is going to build a snapshot that detects changes in products by comparing that date against previous runs — it needs a date column from the data itself (never the system's clock) to work, so this guide adds it starting now, with the same fixed value (2026-08-01, one day before orders's week starts) across all four products: all of them entered the catalog on the same day, in this first version. The _v1 suffix in the file name isn't a coincidence either — it anticipates that, in module 5, a products_v2.csv is going to exist with a real change to unit_cost and category in at least one product.

Worked example (continued): the first sources.yml

With the four files already in raw_data/kiosko/, create the declaration file inside models/staging/kiosko/ — the same folder where, a few lessons from now, the staging models are going to live:

mkdir -p models/staging/kiosko
# models/staging/kiosko/_sources.yml
version: 2

sources:
  - name: kiosko_raw
    description: "Kiosko's raw files, read directly from disk with dbt-duckdb."
    tables:
      - name: orders
        description: "One row per point-of-sale sale, one CSV file per day of the week from 2026-08-03 to 2026-08-09."
      - name: events
        description: "The delivery app's clickstream (page_view, add_to_cart, purchase), one JSON Lines file per day of the same week."
      - name: stores
        description: "Catalog of Kiosko's three stores."
      - name: products
        description: "Product catalog, version 1 (before the price and category change module 5 introduces)."

Three decisions in this file deserve an explanation:

  • The file name starts with an underscore (_sources.yml). It's not a technical dbt requirement — you could call it just sources.yml and it would work the same — it's a convention you're going to see repeated in real dbt projects: configuration files (no SQL logic) get prefixed with _ so, alphabetically, they show up first in any listing of the folder, visually separated from the .sql files that do contain transformations.
  • name: kiosko_raw is the name of the group of sources, not the name of any table. You're always going to use it together with a specific table name — source('kiosko_raw', 'orders') — never alone. Grouping like this makes sense when, as here, several raw tables come from the same source system (in this case, the files Kiosko generates).
  • description: on each table isn't decorative. These descriptions are the first real content you're going to see show up in module 7's autogenerated documentation (dbt docs generate) — writing them now, while the context is fresh, is cheaper than rebuilding them later.

Confirm dbt recognizes all four sources, even with no model yet using them:

dbt ls --select source:kiosko_raw

What to expect.

Running with dbt=1.12.2
Registered adapter: duckdb=1.11.0
[WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
There are 1 unused configuration paths:
- models.kiosko_analytics
Found 4 sources, 500 macros
source:kiosko_analytics.kiosko_raw.events
source:kiosko_analytics.kiosko_raw.orders
source:kiosko_analytics.kiosko_raw.products
source:kiosko_analytics.kiosko_raw.stores

The four sources show up, sorted alphabetically by dbt. The WARNING about "unused configuration paths" is expected and benign at this exact point in the module: dbt_project.yml still says models: kiosko_analytics: +materialized: view from module 1, but since models/ is completely empty — you deleted example/ in step 0, and haven't written any new .sql yet — dbt has nothing to apply that config to. The warning goes away on its own in lesson 7, as soon as at least one real model exists.

Diagram: what exists at the end of this lesson

kiosko_analytics/
├── dbt_project.yml
├── profiles.yml
├── raw_data/
│   └── kiosko/
│       ├── orders_2026-08-03.csv ... orders_2026-08-09.csv   (7 files, 40 rows)
│       ├── events_2026-08-03.jsonl ... events_2026-08-09.jsonl (7 files, 32 rows)
│       ├── stores.csv                                         (3 rows)
│       └── products_v1.csv                                    (4 rows)
└── models/
    └── staging/
        └── kiosko/
            └── _sources.yml   <- 4 sources declared, WITHOUT meta.external_location yet

Notice what does not exist yet: no .sql file, and _sources.yml with no meta key. Both pieces arrive in the next two lessons — lesson 3 explains why source() matters even before you can read a physical file, and lesson 4 finally connects each source to its real file on disk.

Common mistakes

Putting the raw files inside seeds/ "because they're CSVs." What happens: someone, guided by the intuition that "every CSV goes in seeds/," copies orders_2026-08-03.csv there instead of into raw_data/kiosko/. Why it happens: dbt doesn't visually distinguish between a CSV meant for dbt seed and one meant for external_location — both are, literally, the same type of file. How to spot it: if at some point in this project you run dbt seed and see it try to create a table called orders_2026_08_03 (with underscores, because dbt sanitizes the file name), that's the sign a raw file ended up in the wrong place. How to fix it: files you're going to read with source() + external_location always go in a folder outside the *-paths dbt_project.yml declares — raw_data/ in this guide, but the exact name matters less than that it's not seeds/.

Writing sources: as if it were a flat list of tables, without the name: kiosko_raw level. What happens: someone, in a hurry, writes something like sources: [orders, events, stores, products] directly, without the nested name / tables structure. Why it happens: dbt's YAML syntax for sources has two required levels (the group, and the tables inside the group), and it's easy to forget the first one if you've never seen it before. How to spot it: dbt ls --select source:kiosko_raw returns Found 0 sources instead of 4, or dbt parse fails outright with a YAML schema validation error. How to fix it: copy this lesson's worked example's structure exactly — sources: is a list where each element has name (the group) and tables (the list of tables inside that group), never a flat list of loose table names.

Expecting dbt ls --select source:kiosko_raw to confirm the data is readable. What happens: someone sees this lesson's command's four output lines and assumes they can already query orders from somewhere. Why it happens: the command does confirm dbt "knows" the source exists — it's easy to confuse that with "dbt can already read it." How to spot it: if you try writing a model that selects from {{ source('kiosko_raw', 'orders') }} at this point in the module, it's going to fail (you're going to see the exact error in lesson 4) — dbt ls only validates the YAML's syntax, not the existence of a readable physical file. How to fix it: remember the receiving-dock analogy's distinction: logging something in the receiving register (this lesson) is a different step from pointing at the truck's exact address (meta.external_location, lesson 4). Neither one, on its own, moves a single box.

Exercises

Exercise 1 — Verify the file count with your own system. After creating this lesson's files (7 for orders plus 4 for stores/products... it's actually 16 raw files total, counting events), confirm with a terminal command that raw_data/kiosko/ contains exactly 16 files.

See solution 
ls raw_data/kiosko/ | wc -l

The result should be 16: seven orders_*.csv files, seven events_*.jsonl files, stores.csv, and products_v1.csv. If the number doesn't match, check with ls raw_data/kiosko/ which one is missing or extra before continuing — a missing file at this point translates, later, into an incorrect row count in the corresponding staging model.

Exercise 2 — Add the missing description. The worked example's _sources.yml doesn't include a description: at the column level, only at the table level. Check the official sources documentation linked in Resources and add a description for the orders table's order_ts column, without breaking the rest of the file's syntax.

See solution 
tables:
  - name: orders
    description: "One row per point-of-sale sale, one CSV file per day of the week from 2026-08-03 to 2026-08-09."
    columns:
      - name: order_ts
        description: "Date and time the sale was recorded at the point of sale, in the store's local time."

The columns: key accepts the same nested structure you already used for tables: — a list where each element has name and, optionally, description. You're going to come back to this exact structure, with a lot more detail, in module 4, when columns: is also where you declare data_tests: per column.

Exercise 3 — Explain this lesson's boundary in your own words. In 2-3 sentences, using the receiving-dock analogy, explain why dbt ls --select source:kiosko_raw can confirm the four sources "exist" for dbt, without dbt having read a single row of any real file yet.

See solution

_sources.yml is, at this point, just the receiving register with the names logged — dbt knows a group called kiosko_raw exists with four tables called orders, events, stores, and products, the same way a warehouse can have a register with a supplier's name and what merchandise it expects from them, without that truck having arrived yet. dbt ls only validates the YAML is well formed and those names exist as a declaration — it opens no connection to the physical file and runs no SELECT, so it can't (and doesn't try to) confirm the data behind those names is readable.

Summary and next step

In this lesson you cleaned up module 1's trivial model, decided where Kiosko's raw files live — raw_data/kiosko/, deliberately outside seeds/ — and recreated all four files in full: seven days of orders (40 rows), seven days of events (32 rows), stores, and products_v1 (with the new product_updated_at column). You wrote the guide's first sources.yml, declaring the kiosko_raw group with its four tables, and confirmed with dbt ls that dbt already recognizes them as a valid part of the project.

Before moving on you should be able to: explain why the raw files don't go in seeds/; and write from memory the minimal two-level structure (the group's name, tables inside it) sources: syntax requires.

Lesson 3 stops to look at the source() function itself — the one you're going to write inside every SQL model — and at what your project gains, in concrete, measurable terms, from using it instead of writing a file's path directly in a FROM.

Resources

  • dbt Developer Hub — "About sources," the official reference for the complete YAML syntax for declaring sources, including this lesson's name/description/tables/columns structure. docs.getdbt.com/docs/build/sources. In English.
  • dbt Developer Hub — "About seeds," the official reference for dbt seed, useful to confirm by contrast why Kiosko's raw files don't belong there. docs.getdbt.com/docs/build/seeds. In English.
  • dbt Developer Hub — "dbt ls," the reference for the command used in this lesson to list declared resources without running them. docs.getdbt.com/reference/commands/list. In English.
← PreviousModule introduction: from a toy model to Kiosko's real dataNext →The `source()` function and why it matters