Data Engineering interview question · Most asked
How do you monitor data pipelines beyond job success?
Summary
Monitor the data: freshness (age of newest data against an SLO), volume (rows versus history and day-of-week patterns), completeness and distributions (null rates, key metrics), schema changes, and test results. Route alerts to dataset owners with runbooks and block publishing on critical failures.
Longer Answer
Jobs succeed on empty files, stale sources and NULL columns. Data SLOs agreed with consumers turn monitoring into commitments; dashboard banners showing "data delayed" keep users informed.
Key Points
- Freshness, volume, distributions, schema.
- Data SLOs with owners.
- Block publishing on critical failures.
- Communicate delays to users.
Deep dive: ETL and Data Pipelines: Monitoring Freshness, Volume and Data SLOs
Explanation
What interviewers look for
Data-level monitors and ownership.
More most-asked Data Engineering questions
- Batch or streaming: how do you decide?
- What is the difference between ETL and ELT, and when would you choose each?
- What is change data capture (CDC), and why is log-based CDC preferred over polling?
- What does it mean for a data pipeline to be idempotent, and how do you achieve it?
- How do you implement incremental extraction, and what can go wrong?
- How do you deduplicate events, and what does 'exactly-once' really mean?
- What is a star schema, and why is it used in data warehouses?
- What is the grain of a table, and what is join fan-out?