Data Engineering interview question · Most asked
Why use an orchestrator like Airflow or Dagster instead of cron?
Summary
Orchestrators run tasks when their dependencies succeed (not at fixed times), retry failures, stop downstream tasks and alert on errors, track runs by logical date for easy reruns and backfills, limit concurrency, and provide history and a UI.
Longer Answer
Cron time gaps assume upstream steps finish in time; on slow or failed nights downstream steps silently process partial or stale data. Keep business logic in tested tasks (SQL, dbt, Python) and keep the orchestrator thin. Data-aware scheduling (Airflow datasets/assets, Dagster assets) triggers pipelines when inputs update.
Key Points
- Dependency-driven, not clock-driven.
- Retries, alerts, stop on failure.
- Logical dates make backfills routine.
- Keep orchestration thin.
Deep dive: ETL and Data Pipelines: Orchestration
Explanation
What interviewers look for
Concrete failure modes of cron and orchestrator features that fix them.
More most-asked Data Engineering questions
- Batch or streaming: how do you decide?
- What is the difference between ETL and ELT, and when would you choose each?
- What is change data capture (CDC), and why is log-based CDC preferred over polling?
- What does it mean for a data pipeline to be idempotent, and how do you achieve it?
- How do you implement incremental extraction, and what can go wrong?
- How do you deduplicate events, and what does 'exactly-once' really mean?
- What is a star schema, and why is it used in data warehouses?
- What is the grain of a table, and what is join fan-out?