https://community.databricks.com/t5/get-started-discussions/concurrent-state-update-from-worker-nodes-possible/m-p/74868#M3148 <P>For a data processing pipeline I use structured streaming and arbitrary stateful processing. I was wondering if the partitioning over several worker nodes and thus updating the state from different worker nodes has to be considered (e.g. using a lock) when using&nbsp;<SPAN>applyInPandasWithState. Or is that handled automatically by PySpark and Databricks and abstracted away?</SPAN></P><P>Thank you</P> Tue, 18 Jun 2024 13:42:44 GMT fperry 2024-06-18T13:42:44Z https://community.databricks.com/t5/get-started-discussions/concurrent-state-update-from-worker-nodes-possible/m-p/74868#M3148 <P>For a data processing pipeline I use structured streaming and arbitrary stateful processing. I was wondering if the partitioning over several worker nodes and thus updating the state from different worker nodes has to be considered (e.g. using a lock) when using&nbsp;<SPAN>applyInPandasWithState. Or is that handled automatically by PySpark and Databricks and abstracted away?</SPAN></P><P>Thank you</P> Tue, 18 Jun 2024 13:42:44 GMT https://community.databricks.com/t5/get-started-discussions/concurrent-state-update-from-worker-nodes-possible/m-p/74868#M3148 fperry 2024-06-18T13:42:44Z