Data Engineering interview question · Most asked
How do you handle late-arriving data in a daily batch pipeline?
Summary
Partition by event time and recompute a lookback window (for example the last 3 days) on every run with idempotent partition overwrites, sizing the window from measured arrival delays. Label recent numbers provisional, run wider periodic corrections, and use adjustment records where published numbers must not change.
Longer Answer
Event time (when it happened) differs from arrival time, especially for mobile devices, partners and retried webhooks. Measure the delay distribution and choose a window that covers the vast majority; rare stragglers can be handled by weekly corrections or a late-arrivals table.
Key Points
- Event time vs arrival time.
- Recompute a lookback window idempotently.
- Size it from measured delays.
- Provisional vs final numbers.
Deep dive: ETL and Data Pipelines: Late and Out-of-Order Data
Explanation
What interviewers look for
Event-time partitioning, lookback recomputation and communication of provisional data.
More most-asked Data Engineering questions
- Batch or streaming: how do you decide?
- How do you implement incremental extraction, and what can go wrong?
- What is change data capture (CDC), and why is log-based CDC preferred over polling?
- What does it mean for a data pipeline to be idempotent, and how do you achieve it?
- How do you deduplicate events, and what does 'exactly-once' really mean?
- What is a star schema, and why is it used in data warehouses?
- What is the grain of a table, and what is join fan-out?
- What is the difference between ETL and ELT, and when would you choose each?