Databricks Community

jonxu · 3 weeks ago

Any one can help me understand why cannot we unify streaming with batch and unbounded with bounded, if we regard the streaming/unbounded as mini-version of batch/bounded, please?

i.e., if I set one second as the frequency for batch processing, will it be the same as streaming, please?

VZLA · 2 weeks ago

This could be a very long thread, but to summarize it we could refer to the improvements or streaming specific features to understand why these two are not the same, and why these features were created. Essentialy, Spark streaming(unbounded) jobs are fundamentally different from high-frequency batch jobs(bounded), so it’s not as simple as treating streaming as "tiny batches". Why?:

Structured Streaming API: Spark has an API just for streaming, designed to handle continuous, real-time data without waiting for entire datasets like batch jobs do. Reaching this level of improvement and difference by self implementing them in a batch job, would require a lot of end user job (not worth it)
Micro-Batch Execution: Structured Streaming processes data in near-instant “micro-batches,” starting processing as soon as data arrives, which a high-frequency batch job can’t do effectively.
Event Time & Watermarking: Unlike batch jobs, Spark streaming can work with event time and uses watermarks to handle late data and out-of-order events accurately.
Stateful Processing: Streaming can track and hold state across micro-batches for long-running aggregations, which batch jobs lack without constant workarounds.
Continuous Processing Mode: Spark offers experimental continuous processing for true real-time execution, which batch jobs can’t replicate since they rely on periodic scheduling.
Exactly-Once Fault Tolerance: Streaming jobs use checkpoints to achieve exactly-once processing, ensuring data accuracy even if a failure happens mid-stream.
Trigger Control: In streaming, triggers manage when to process data based on data arrival, while batch jobs would require custom scheduling logic to even come close.

So, streaming isn’t exactly “mini-batch” — it’s purpose-built for handling unbounded data with real-time constraints.

https://youtu.be/i31QggFi4mo?feature=shared&t=794

Hope it helps.

View solution in original post

VZLA · 2 weeks ago

This could be a very long thread, but to summarize it we could refer to the improvements or streaming specific features to understand why these two are not the same, and why these features were created. Essentialy, Spark streaming(unbounded) jobs are fundamentally different from high-frequency batch jobs(bounded), so it’s not as simple as treating streaming as "tiny batches". Why?:

Structured Streaming API: Spark has an API just for streaming, designed to handle continuous, real-time data without waiting for entire datasets like batch jobs do. Reaching this level of improvement and difference by self implementing them in a batch job, would require a lot of end user job (not worth it)
Micro-Batch Execution: Structured Streaming processes data in near-instant “micro-batches,” starting processing as soon as data arrives, which a high-frequency batch job can’t do effectively.
Event Time & Watermarking: Unlike batch jobs, Spark streaming can work with event time and uses watermarks to handle late data and out-of-order events accurately.
Stateful Processing: Streaming can track and hold state across micro-batches for long-running aggregations, which batch jobs lack without constant workarounds.
Continuous Processing Mode: Spark offers experimental continuous processing for true real-time execution, which batch jobs can’t replicate since they rely on periodic scheduling.
Exactly-Once Fault Tolerance: Streaming jobs use checkpoints to achieve exactly-once processing, ensuring data accuracy even if a failure happens mid-stream.
Trigger Control: In streaming, triggers manage when to process data based on data arrival, while batch jobs would require custom scheduling logic to even come close.