Concurrent State Update from Worker Nodes Possible?

[fperry]

For a data processing pipeline I use structured streaming and arbitrary stateful processing. I was wondering if the partitioning over several worker nodes and thus updating the state from different worker nodes has to be considered (e.g. using a lock) when using applyInPandasWithState. Or is that handled automatically by PySpark and Databricks and abstracted away?

Thank you

[Kaniz_Fatma]

Hi @fperry, When using applyInPandasWithState in PySpark, updates to each group’s state are automatically saved ...¹. The function you provide should take parameters (key, Iterator[pandas.DataFrame], state) and return another Iterator[pandas.DataFrame]. The grouping key(s) will be passed as a tuple of numpy data types (e.g., numpy.int32 and numpy.float64). As for partitioning and state updates across worker nodes, PySpark and Databricks handle this abstraction for you, so you don’t need to manually manage locks or synchronization mechanisms. 😊