Community-produced videos to help you leverage Databricks in your Data & AI journey. Tune in to explore industry trends and real-world use cases from leading data practitioners.
In this video, Craig Lukasik, a Senior Specialist Solutions Architect at Databricks, will cover state schema evolution in streaming. Delta Lake handles schema evolution. But what if your state which is used in stateful Structured Streaming needs to evolve? This video helps you understand the nuances of schemas in stateful Structured Streaming and provides a strategy for evolving state schema. The focus is on PySpark and the applyInPandasWithState operator. applyInPandas allows users to perform intricate operations while preserving the state. This is invaluable when dealing with multiple records from different streams. The video also goes over a detailed demo including data generation, building pipelines using the medallion architecture and the use of applyInPandas. Craig drops a ton of tips along the way, so make sure you watch the video in entirety!
