Topic hub
ETL and Data Pipelines
Moving data reliably from production systems to warehouses: ETL vs ELT, change data capture, idempotent loads, late data, dimensional modelling, dbt, orchestration, data quality and cost, with interview questions and a quiz.
Lessons
- Transactional Outbox and Change Data Capture · Messaging, Events and Email Systems
Publish events reliably together with database changes. - Change Data Capture · Distributed Systems for Backend Engineers
Stream database changes to other systems reliably. - Batch, Micro-Batch or Streaming? · ETL and Data Pipelines
How a vague request for 'real-time' data leads to an expensive streaming platform nobody needed, and how to choose a pipeline style from a measurable freshness requirement. - Change Data Capture for Pipelines · ETL and Data Pipelines
Why polling tables misses deletes and overloads the primary database, and how log-based change data capture streams every insert, update and delete in commit order. - ETL vs ELT: Where the Transform Happens · ETL and Data Pipelines
Why a nightly job that transforms data before loading it gets slower every month and can't answer new questions, and when loading raw data first (ELT) is the better shape. - Incremental Extraction and Watermarks · ETL and Data Pipelines
Why 'WHERE updated_at > last run' silently misses rows, and how overlap windows, idempotent loads and commit-ordered positions make incremental extraction safe. - Deduplication and the Exactly-Once Myth · ETL and Data Pipelines
Why event counts were inflated by retried deliveries, and how to deduplicate with keys and window functions instead of hoping for exactly-once delivery. - Idempotent Loads: Safe to Run Twice · ETL and Data Pipelines
Why rerunning a failed job doubled last Tuesday's revenue, and how upserts and partition overwrites make every load safe to repeat. - Late and Out-of-Order Data · ETL and Data Pipelines
Why yesterday's totals keep changing after the report was sent, and how event time, lookback windows and 'provisional' numbers handle data that arrives late. - Slowly Changing Dimensions · ETL and Data Pipelines
Why last year's sales by region changed when a customer moved city, and how Type 2 dimensions keep history so past reports stay true. - Star Schemas and Grain · ETL and Data Pipelines
Why revenue doubled after someone joined orders to order lines, and how declaring the grain of every table and modelling facts and dimensions prevents fan-out errors. - Managing SQL Transformations with dbt · ETL and Data Pipelines
Why a folder of SQL scripts run by cron produced stale numbers, and how dbt builds a dependency graph from ref(), tests models and makes transformations reviewable. - Data Quality Checks and Data Contracts · ETL and Data Pipelines
How a renamed column upstream turned revenue into zero while every job stayed green, and how automated checks and data contracts catch bad data before people do. - Monitoring Freshness, Volume and Data SLOs · ETL and Data Pipelines
Why a pipeline 'succeeded' every night for a week while loading zero rows, and how freshness, volume and distribution monitors with clear owners catch silent failures. - Orchestration: Dependencies, Not Clock Times · ETL and Data Pipelines
Why a cron schedule that 'leaves an hour of buffer' produced half-empty reports on busy days, and how an orchestrator runs steps when their inputs are ready.