Batch or streaming: how do you decide?

Summary

Start from a freshness requirement agreed with the data's consumers, then choose the simplest style that meets it: daily or hourly batch, micro-batch every few minutes, or true streaming only when decisions need seconds (fraud checks, live pricing, alerting).

Longer Answer

Streaming adds always-on infrastructure, state and checkpoint management, harder reprocessing, late-event handling and more on-call load. Many platforms combine both: a stream for a few low-latency use cases and batch for everything else, often fed from the same change data capture feed.

Key Points

• Turn 'real-time' into a freshness target.
• Prefer batch, then micro-batch, then streaming.
• Streaming costs complexity and on-call load.
• Mixed architectures are common.

Deep dive: ETL and Data Pipelines: Batch, Micro-Batch or Streaming?

What interviewers look for

A requirement-driven decision and awareness of streaming's operational cost.