https://community.databricks.com/t5/databricks-tv/state-schema-evolution-in-pyspark-using-applyinpandaswithstate/ba-p/98928 <P><IFRAME src="https://www.youtube.com/embed/qZdViHE_1U8?si=WBn-HtgnEmk9n2cM" width="560" height="315" frameborder="0" allowfullscreen="" title="YouTube video player" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin"></IFRAME></P> <P><SPAN>In this video, Craig Lukasik, a Senior Specialist Solutions Architect at Databricks, will cover state schema evolution in streaming. Delta Lake handles schema evolution. But what if your state which is used in stateful Structured Streaming needs to evolve? This video helps you understand the nuances of schemas in stateful Structured Streaming and provides a strategy for evolving state schema. The focus is on PySpark and the applyInPandasWithState operator. applyInPandas allows users to perform intricate operations while preserving the state. This is invaluable when dealing with multiple records from different streams. The video also goes over a detailed demo including data generation, building pipelines using the medallion architecture and the use of applyInPandas. Craig drops a ton of tips along the way, so make sure you watch the video in entirety!</SPAN></P> Mon, 27 Jan 2025 14:09:18 GMT Adyasha 2025-01-27T14:09:18Z https://community.databricks.com/t5/databricks-tv/state-schema-evolution-in-pyspark-using-applyinpandaswithstate/ba-p/98928 <P><STRONG>Master Schema Evolution in Stateful Streaming with PySpark</STRONG></P> <P>Join Craig Lukasik, Senior Solutions Architect at Databricks, as he explores state schema evolution in stateful Structured Streaming. Learn how to manage schema changes using PySpark and the <CODE>applyInPandasWithState</CODE> operator, preserving state across streams. Watch a detailed demo on building pipelines with the medallion architecture and evolving state schemas. Packed with expert tips, this video is essential for mastering schema evolution in streaming!</P> Mon, 27 Jan 2025 14:09:18 GMT https://community.databricks.com/t5/databricks-tv/state-schema-evolution-in-pyspark-using-applyinpandaswithstate/ba-p/98928 Adyasha 2025-01-27T14:09:18Z