Data Engineering interview question · Most asked
Scenario: a nightly job ran successfully but the dashboard shows zero sales for yesterday. What happened and how do you prevent it?
Summary
Likely an empty or partial input (empty export, upstream outage), a renamed or missing column loaded as NULL, or a filter/date bug. Prevent it with volume and freshness monitors, schema and not-null checks on key measures, inputs that fail loudly when empty, and blocking publication on critical test failures.
Longer Answer
Job success only means the code ran. Add data SLOs with owners and show 'data delayed' on dashboards when checks fail.
Key Points
- Empty inputs and schema changes don't error.
- Volume, freshness and null checks.
- Fail loudly on empty inputs.
- Block publication on critical failures.
Deep dive: ETL and Data Pipelines: Monitoring Freshness, Volume and Data SLOs
Explanation
What interviewers look for
Data-level monitoring rather than job-level.
More most-asked Data Engineering questions
- Batch or streaming: how do you decide?
- What is the difference between ETL and ELT, and when would you choose each?
- What is change data capture (CDC), and why is log-based CDC preferred over polling?
- What does it mean for a data pipeline to be idempotent, and how do you achieve it?
- How do you implement incremental extraction, and what can go wrong?
- How do you deduplicate events, and what does 'exactly-once' really mean?
- What is a star schema, and why is it used in data warehouses?
- What is the grain of a table, and what is join fan-out?