โ11-03-2024 03:21 PM
Any one can help me understand why cannot we unify streaming with batch and unbounded with bounded, if we regard the streaming/unbounded as mini-version of batch/bounded, please?
i.e., if I set one second as the frequency for batch processing, will it be the same as streaming, please?
โ11-04-2024 04:44 AM - edited โ11-04-2024 04:47 AM
This could be a very long thread, but to summarize it we could refer to the improvements or streaming specific features to understand why these two are not the same, and why these features were created. Essentialy, Spark streaming(unbounded) jobs are fundamentally different from high-frequency batch jobs(bounded), so itโs not as simple as treating streaming as "tiny batches". Why?:
Structured Streaming API: Spark has an API just for streaming, designed to handle continuous, real-time data without waiting for entire datasets like batch jobs do. Reaching this level of improvement and difference by self implementing them in a batch job, would require a lot of end user job (not worth it)
Micro-Batch Execution: Structured Streaming processes data in near-instant โmicro-batches,โ starting processing as soon as data arrives, which a high-frequency batch job canโt do effectively.
Event Time & Watermarking: Unlike batch jobs, Spark streaming can work with event time and uses watermarks to handle late data and out-of-order events accurately.
Stateful Processing: Streaming can track and hold state across micro-batches for long-running aggregations, which batch jobs lack without constant workarounds.
Continuous Processing Mode: Spark offers experimental continuous processing for true real-time execution, which batch jobs canโt replicate since they rely on periodic scheduling.
Exactly-Once Fault Tolerance: Streaming jobs use checkpoints to achieve exactly-once processing, ensuring data accuracy even if a failure happens mid-stream.
Trigger Control: In streaming, triggers manage when to process data based on data arrival, while batch jobs would require custom scheduling logic to even come close.
So, streaming isnโt exactly โmini-batchโ โ itโs purpose-built for handling unbounded data with real-time constraints.
https://youtu.be/i31QggFi4mo?feature=shared&t=794
Hope it helps.
โ11-04-2024 04:44 AM - edited โ11-04-2024 04:47 AM
This could be a very long thread, but to summarize it we could refer to the improvements or streaming specific features to understand why these two are not the same, and why these features were created. Essentialy, Spark streaming(unbounded) jobs are fundamentally different from high-frequency batch jobs(bounded), so itโs not as simple as treating streaming as "tiny batches". Why?:
Structured Streaming API: Spark has an API just for streaming, designed to handle continuous, real-time data without waiting for entire datasets like batch jobs do. Reaching this level of improvement and difference by self implementing them in a batch job, would require a lot of end user job (not worth it)
Micro-Batch Execution: Structured Streaming processes data in near-instant โmicro-batches,โ starting processing as soon as data arrives, which a high-frequency batch job canโt do effectively.
Event Time & Watermarking: Unlike batch jobs, Spark streaming can work with event time and uses watermarks to handle late data and out-of-order events accurately.
Stateful Processing: Streaming can track and hold state across micro-batches for long-running aggregations, which batch jobs lack without constant workarounds.
Continuous Processing Mode: Spark offers experimental continuous processing for true real-time execution, which batch jobs canโt replicate since they rely on periodic scheduling.
Exactly-Once Fault Tolerance: Streaming jobs use checkpoints to achieve exactly-once processing, ensuring data accuracy even if a failure happens mid-stream.
Trigger Control: In streaming, triggers manage when to process data based on data arrival, while batch jobs would require custom scheduling logic to even come close.
So, streaming isnโt exactly โmini-batchโ โ itโs purpose-built for handling unbounded data with real-time constraints.
https://youtu.be/i31QggFi4mo?feature=shared&t=794
Hope it helps.
โ11-04-2024 01:13 PM
Many thanks for the clarification!
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group