Orchestration: Dependencies, Not Clock Times

Scenario

Example company: three cron jobs run each night: extract at 01:00, transform at 02:00 ("extract never takes more than 40 minutes") and publish reports at 03:00.

What you see in production

• After a sale weekend, extract takes 75 minutes. Transform starts at 02:00 on a half-loaded table, and the reports emailed at 03:00 show about half of Sunday's revenue.
• When extract fails at 01:10, transform and publish still run, on yesterday's data, and nobody is alerted.
• Re-running "just the transform for Tuesday" means editing crontab and scripts by hand.

The naive code

0 1 * * *  extract.sh
0 2 * * *  transform.sh      # assumes extract finished
0 3 * * *  publish.sh        # assumes transform finished

Reproduce it

The simulation runs four nights with varying extract times, scheduled by clock time and by dependencies.

nights = {"Mon": 40, "Tue": 45, "Wed": 75, "Thu": None}    # extract minutes; None = extract failed

for night, extract_min in nights.items():
    # cron: transform always starts 60 minutes after extract started
    if extract_min is None:
        cron = "transform + publish ran on yesterday's data (no alert)"
    elif extract_min > 60:
        cron = f"transform started at 60 min, extract still running until {extract_min} -> partial report"
    else:
        cron = "ok"
    # orchestrator: transform waits for extract to succeed; failure stops downstream and alerts
    if extract_min is None:
        dag = "extract failed -> transform and publish skipped, on-call alerted, retry scheduled"
    else:
        dag = f"transform started at {extract_min} min, publish after transform -> complete report"
    print(f"{night}: cron: {cron}\n     DAG:  {dag}")

Mon: cron: ok
     DAG:  transform started at 40 min, publish after transform -> complete report
Tue: cron: ok
     DAG:  transform started at 45 min, publish after transform -> complete report
Wed: cron: transform started at 60 min, extract still running until 75 -> partial report
     DAG:  transform started at 75 min, publish after transform -> complete report
Thu: cron: transform + publish ran on yesterday's data (no alert)
     DAG:  extract failed -> transform and publish skipped, on-call alerted, retry scheduled

Root cause

Dependencies were encoded as clock-time gaps. Clock gaps can't know whether the upstream step finished, succeeded or produced data, so every slow or failed night breaks the chain silently.

The fix

Use an orchestrator that models the pipeline as a graph of tasks with explicit dependencies:

• Apache Airflow DAGs, Dagster assets, Prefect flows, or managed services (AWS Step Functions, Google Cloud Composer, Azure Data Factory).
• Downstream tasks run when upstream tasks succeed, not at a fixed time.
• Retries with backoff for transient failures; failures stop downstream tasks and alert.
• Each run has a logical date (the data interval it processes), so reruns and backfills of a specific day are one command.
• Cross-pipeline dependencies can be data-driven: Airflow's data-aware scheduling (datasets, called assets in Airflow 3) and Dagster's asset graph start a pipeline when the data it needs is updated.
• Use sensors sparingly to wait for external files or partitions, with timeouts.

Keep business logic in the tasks (SQL, dbt, Python modules) and the orchestrator thin: it decides when and in what order, not what.

Trade-offs

• An orchestrator is another system to run, upgrade and secure; managed versions reduce that.
• Very simple pipelines (one job) can stay on cron if failures alert and the job is idempotent.

Detect it in production

• Alert on task failure, on retries exhausted, and on SLA misses (a run not finished by a deadline).
• Track run duration per task to see growth before it breaks deadlines.

Key points

• Express dependencies as edges in a graph, never as clock-time gaps.
• Failures should stop downstream steps and alert.
• Runs carry a logical date, so reruns and backfills are routine.
• Keep the orchestrator thin; logic lives in tested tasks.