DATA ENGINEERING · QUESTION 26
What is the difference between batch and stream processing?
On this page
Batch processing collects a bounded set of records and processes them together on a schedule or trigger. Stream processing handles events continuously or in small windows as they arrive, supporting fresher results with more operational complexity. The choice depends on freshness requirements, event volume, ordering, late data, and cost. Many systems combine both patterns for different outputs or recovery workflows.
LET'S MAKE IT SIMPLE
First, the idea in plain English
A report that runs each morning can process yesterday’s completed orders as one batch. A fraud alert may need to react as each payment arrives. Batch work is often simpler to rerun and reason about; streaming can reduce delay but must deal with events arriving late, duplicated, or out of order. Neither is always better—the business need for freshness should justify the extra complexity.
A USEFUL COMPARISON
Picture it this way
Batch processing is like collecting a day’s mail and sorting it in one sitting. Stream processing is like sorting each letter as it arrives. The second gets information sooner, but needs a process that keeps running and handles interruptions. Real event systems also need to define time, duplicates, ordering, and recovery.
FOLLOW THE JOURNEY
Compare scheduled batches with continuous event flows
Show how the same incoming records are grouped in batch processing or handled as a live stream.
Streaming systems may use event-time windows and still wait for late data; “streaming” does not guarantee instant or complete results.
Events enter either a scheduled batch that processes a bounded group or a streaming processor that handles events continuously and emits updated results.
How to read this visual
Streaming systems may use event-time windows and still wait for late data; “streaming” does not guarantee instant or complete results.
UNDER THE HOOD
How it works, step by step
Set the freshness target
Determine how quickly consumers need an updated result.
Why it matters: Freshness often drives whether continuous processing is worthwhile.
Understand arrival behavior
Check event rates, late arrivals, duplicates, and ordering requirements.
Why it matters: These properties influence correctness and state management.
Choose processing mode
Use scheduled batches, streaming, or a combination based on the target and complexity.
Why it matters: The architecture should solve the requirement without unnecessary operations.
Define replay and recovery
Specify checkpoints, offsets, idempotency, and how corrected events update outputs.
Why it matters: Failures and late records must not silently corrupt results.
OUTSIDE THE TEXTBOOK
A practical example
A retailer may process product catalog changes in a nightly batch because daily freshness is enough, while monitoring payment fraud in a stream because action must happen during checkout. Both flows can land into a common analytical store, but their freshness, late-data, and retry rules differ.
GO A LITTLE DEEPER
Technical details that matter
- Event time is when an event occurred; processing time is when the system handled it. Windowing by event time can improve correctness for delayed records.
- Exactly-once effects require careful coordination between source offsets, processing state, and output commits; the phrase is not a blanket guarantee.
- A micro-batch engine processes small bounded groups frequently and can combine operational traits of batch and streaming.
- Backpressure, state size, checkpointing, and replay retention are important operational considerations for continuous jobs.
MAKE A GOOD CHOICE
When is this useful?
- Batch for scheduled reports, historical recomputation, and workloads where minutes or hours of delay are acceptable.
- Streaming for alerts, live dashboards, or decisions that need low-latency event updates.
- Hybrid architectures where real-time outputs need periodic reconciliation or backfills.
A frequent scheduled job may meet a modest freshness target with simpler operations than streaming. Conversely, if action must occur before a user request completes, a batch pipeline is likely too slow.
WATCH OUT FOR
Common misunderstandings
- Choosing streaming because it sounds more modern when the freshness requirement is loose.
- Assuming streaming means events arrive in order exactly once.
- Ignoring backfills and late data when designing only for the happy path.
REAL-WORLD TRADE-OFFS
Limitations and failure cases
- Streaming adds continuously running infrastructure, state, and incident-response demands.
- Batch results can be stale between runs and large reruns can consume resources.
- Combining both patterns can create reconciliation and consistency complexity.
WHAT TO REMEMBER
Key takeaways
- 1Batch groups bounded data; streaming handles events continuously or in windows.
- 2Freshness requirements should lead the choice.
- 3Late, duplicate, and out-of-order events need explicit policies.
- 4Recovery and replay are part of pipeline correctness.
PRACTICE THE FOLLOW-UP
What might an interviewer ask next?
What is event time versus processing time?
Event time is when the source says something happened; processing time is when the pipeline handled it. They differ when data is delayed or retried.
Can a streaming job process late events?
Yes, with event-time windows and a lateness policy, but the system must define how long to wait and whether prior results can be updated.
When might you choose micro-batching?
It can provide frequent updates using bounded processing units when true per-event handling is unnecessary or operationally costly.
SAY IT OUT LOUD
How to explain this in an interview
“Batch processing works on a bounded group of records, often on a schedule. Stream processing handles events continuously or in short windows for fresher results, but needs more care around state, late events, duplicates, and recovery. I would start with the required freshness and consequences of delay, then choose batch, streaming, or a hybrid that the team can operate reliably.”
Connect this question to the roadmap topic it supports.