Community-produced videos to help you leverage Databricks in your Data & AI journey. Tune in to explore industry trends and real-world use cases from leading data practitioners.
In this video, Vikas Reddy, a Staff Backline Engineer at Databricks, goes over the use and importance of using checkpointing in streaming applications. Asynchronous state checkpointing maintains exactly-once guarantees for streaming queries but can reduce overall latency for some Structured Streaming stateful workloads bottlenecked on state updates. This is accomplished by beginning to process the next micro-batch as soon as the computation of the previous micro-batch has been completed without waiting for state checkpointing to complete. This is explained in a lot more detail (with various examples and demos!) in the 1 hour long video here. Target Audience - Data Engineers, Spark Engineers, Data/Solutions Architects
