Film 1 of 10 · The series and its method
Declare it, then build it
dbt calls every query a model. The data model is the blueprint: declared beside the code, and checked by the tests.
Topics › Analytics engineering › In the weeds of data crafting
In the weeds of data crafting
A technical series for analytics engineers: how an agreed data model becomes tables, with dbt, on Databricks. It follows Jun, the university's analytics engineer, from a question to versioned, tested and documented models, in four phases (ask, promise, build, keep), with an AI agent that helps at every step and a person who approves each one.
The films
Watch them in order. It's technical: it shows real code, YAML and Markdown from an example dbt project that runs on dbt Core with DuckDB, in this site's repository, so you can run it yourself. It makes most sense after From words to data.
Film 2 of 10 · Scope and owners
Start from a question
A model has no edge until a question gives it one. Model only what the question touches, and name who owns each meaning first.
Film 3 of 10 · Identity and keys
What makes it the same one
One learner, three keys, and a look-alike. Rules match most of them; a person decides the rest, and the decision becomes data and a test.
Film 4 of 10 · Grain and time
One row of what, and when
Twelve on census day, nine today: both right. Say what one row is and which day it describes, before any SQL, and test both.
Film 5 of 10 · Contracts and tests
Promises and proofs
Every gap gets a decision. Promise it in a contract; prove it with a test, written before the code it checks.
Film 6 of 10 · Layers and CTEs
Built in layers
Four layers turn the tests green. Each layer has one job; each CTE, one step; and the rule is written once.
Film 7 of 10 · Domains and ownership
Who owns what
Each fact's meaning has an owner. Owners publish a public, versioned core; consumers build on it, never on each other's marts.
Film 8 of 10 · Agents and review
An agent on the team
An AI agent drafts in minutes. It follows rules written in the project, shows the query behind every claim, never weakens a test, and people approve.
Film 9 of 10 · Documentation and versions
Written once
Every copy of a fact is a chance to drift. One home per fact, everything else generated, and a breaking change arrives as a new version with a date.
Film 10 of 10 · The whole process
End to end
One question, ten steps, ten commits. Every file in its place, for as long as it's needed.
One thread: the credential, built
From words to data ends with version 3 of the credential model approved. This series builds it, from one question asked by planning: how many learners are within 15 credit points of a graduate certificate, as at census date? It needs credentials from three systems, one learner keyed three ways, credit that counts towards awards, and history.
Every film opens in the past
| Film | Opens with | Still true today |
|---|---|---|
| Declare it, then build it | Blueprints, the 1870s | A plan says exactly what the building will be, and lays no brick. |
| Start from a question | John Snow's cholera map, Soho, 1854 | Show only what the question needs. |
| What makes it the same one | Bertillon's measurements, Paris, the 1880s | Looking alike isn't being the same. |
| One row of what, and when | The United States census of 1 June 1890 | One card per person, as at one day. |
| Promises and proofs | The hallmark, from 1300 | Tested before it's marked; then trusted without testing again. |
| Built in layers | Escoffier's kitchen brigade, the 1890s | Each station has one job, and every plate is checked at the pass. |
| Who owns what | The Torrens title, South Australia, 1858 | The register is the title: its owner changes it, and everyone relies on it. |
| An agent on the team | The Nautical Almanac, 1766 | Computed twice, and compared, before anything is printed. |
| Written once | One tuning fork for France, 1859 | Kept in one place, and everything else tuned from it. |
| End to end | York Minster's tracing floor, the 1300s | The drawings were for the work; the windows are what stayed. |
Who builds, who decides
Cyan outlines are technical; gold outlines are business.
Builds the model
Jun, analytics engineer, the series' guide
Owns the model
Noor, data architect
Owns the words
Mei, registrar's office
Asks the questions
Planning, the learner's wallet app, and Finance