Dr. Mario's 2nd 🧠

Lessons From the Warehouse

1 min read

rw-book-cover

Metadata

Highlights

Regarding Process Allergy: “structure” is not “process”.

Think about a dbt DAG, and for simplicity, let’s say we’re running 1 thread. We’ve got two unrelated models called customers and and events. Now even though they could, in theory, run in parallel, practically they can’t; one will have to wait until the other is built and tested to start running, because we can only operate on one thread at a time. Our empowered decision makers are like our threads when we create process. Now imagine we throw the thread count wide open to 128 threads, but we implement a set of automated tests, pre-commit hooks, linters, CI/CD, and a formatting tool that enforces our style guide. Now we’ve created more structure, but minimized process. Structure, the actions and elements of a single layer of process, sans the control mechanisms and hierarchy, is a positive force. (View Highlight)

structure is not process, and structure helps more people go faster (View Highlight)

Graph View