How would you decide between batch and streaming for a new dataset?

Agree a freshness target (how old the data may be) with the people who use it, then choose the simplest style that meets it: daily or hourly batch, micro-batch every few minutes, or streaming when decisions need seconds.

What is micro-batching?

Running batch logic on a short schedule (every few seconds to minutes) instead of processing events one by one. It gives near-real-time freshness with batch semantics, which are easier to rerun. Spark Structured Streaming uses micro-batches by default.

What costs does streaming add compared with batch?

Always-on infrastructure, state stores and checkpoints, harder reprocessing (replays), handling of late and out-of-order events, and more on-call load and expertise.

Batch, Micro-Batch or Streaming? — ETL and Data Pipelines

Scenario

Example company: operations asks for a "real-time" stock dashboard. The data team builds a streaming pipeline with a message broker and a stream-processing cluster for every table: orders, customers, products and returns.

What you see in production

The streaming cluster costs more per month than the rest of the data platform combined, and it pages the on-call engineer at night.
Most dashboards are viewed twice a day. The stock dashboard is checked every 15 minutes.
Bugs in streaming jobs are hard to fix: reprocessing old data needs a replay, and two engineers understand the setup.

The naive code

requirement: "real-time"
decision:    stream every table, always

Reproduce it

The model compares freshness and monthly cost for each style, then picks the cheapest one that meets each dataset's freshness need.

Choose the cheapest style that meets the freshness needTry it yourself

styles = {  # worst-case data age (minutes) and rough monthly cost
    "daily batch":       {"age_min": 24 * 60, "cost": 100},
    "hourly batch":      {"age_min": 60,      "cost": 250},
    "15-min micro-batch": {"age_min": 15,     "cost": 600},
    "streaming":         {"age_min": 0.1,     "cost": 4000},
}

needs = {  # agreed freshness requirement per dataset, in minutes
    "finance revenue report": 24 * 60,
    "marketing campaign stats": 60,
    "warehouse stock levels": 15,
    "card fraud signals": 0.5,
}

total_naive = len(needs) * styles["streaming"]["cost"]
total_chosen = 0
for dataset, need in needs.items():
    ok = [(s["cost"], name) for name, s in styles.items() if s["age_min"] <= need]
    cost, name = min(ok)
    total_chosen += cost
    print(f"{dataset:<26} needs <= {need:>6} min -> {name}")

print(f"stream everything: {total_naive}/month | right-sized: {total_chosen}/month")

Output

finance revenue report     needs <=   1440 min -> daily batch
marketing campaign stats   needs <=     60 min -> hourly batch
warehouse stock levels     needs <=     15 min -> 15-min micro-batch
card fraud signals         needs <=    0.5 min -> streaming
stream everything: 16000/month | right-sized: 4950/month

Root cause

"Real-time" was never turned into a number. Without a freshness requirement per dataset, the team designed for the strictest imaginable case everywhere.

The fix

Ask, per dataset: how old can the data be before a decision goes wrong? Write it down as a freshness target (an SLO), for example "stock levels at most 15 minutes old".
Choose the simplest style that meets it:
- Batch (daily or hourly): the default. Simple, cheap, easy to rerun.
- Micro-batch (every few minutes): the same batch code on a short schedule. Spark Structured Streaming also works this way by default.
- Streaming (seconds): only where a decision must happen within seconds, such as fraud checks, live pricing or alerting. Tools include Kafka with Apache Flink or Spark Structured Streaming.
Revisit the target when usage changes.

Trade-offs

Streaming gives seconds-level freshness but adds always-on infrastructure, state management, harder reprocessing and on-call load.
Batch is easy to rerun and reason about, but data is only as fresh as the schedule.
Many teams run both: a stream for the few low-latency use cases, and batch for everything else, often from the same CDC feed.

Detect it in production

Compare each dataset's actual freshness with its target. Large slack means you are over-engineering; misses mean you need a faster style.
Track cost per dataset. Expensive pipelines feeding rarely viewed dashboards are candidates for a slower schedule.

Key points

Turn "real-time" into a freshness target in minutes for each dataset.
Prefer batch, then micro-batch, then streaming, in that order.
Stream only where decisions need seconds.
Measure actual freshness and cost against the targets.