This could be a very long thread, but to summarize it we could refer to the improvements or streaming specific features to understand why these two are not the same, and why these features were created. Essentialy, Spark streaming(unbounded) jobs are fundamentally different from high-frequency batch jobs(bounded), so itโs not as simple as treating streaming as "tiny batches". Why?:
-
Structured Streaming API: Spark has an API just for streaming, designed to handle continuous, real-time data without waiting for entire datasets like batch jobs do. Reaching this level of improvement and difference by self implementing them in a batch job, would require a lot of end user job (not worth it)
-
Micro-Batch Execution: Structured Streaming processes data in near-instant โmicro-batches,โ starting processing as soon as data arrives, which a high-frequency batch job canโt do effectively.
-
Event Time & Watermarking: Unlike batch jobs, Spark streaming can work with event time and uses watermarks to handle late data and out-of-order events accurately.
-
Stateful Processing: Streaming can track and hold state across micro-batches for long-running aggregations, which batch jobs lack without constant workarounds.
-
Continuous Processing Mode: Spark offers experimental continuous processing for true real-time execution, which batch jobs canโt replicate since they rely on periodic scheduling.
-
Exactly-Once Fault Tolerance: Streaming jobs use checkpoints to achieve exactly-once processing, ensuring data accuracy even if a failure happens mid-stream.
-
Trigger Control: In streaming, triggers manage when to process data based on data arrival, while batch jobs would require custom scheduling logic to even come close.
So, streaming isnโt exactly โmini-batchโ โ itโs purpose-built for handling unbounded data with real-time constraints.
https://youtu.be/i31QggFi4mo?feature=shared&t=794
Hope it helps.
