cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
fperry
by • New Contributor II
  • 871 Views
  • 1 replies
  • 0 kudos

Resolved! Concurrent State Update from Worker Nodes Possible?

For a data processing pipeline I use structured streaming and arbitrary stateful processing. I was wondering if the partitioning over several worker nodes and thus updating the state from different worker nodes has to be considered (e.g. using a lock...

Get Started Discussions
Reply
  • 871 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @fperry, When using applyInPandasWithState in PySpark, updates to each group’s state are automatically saved across invocations1. The function you provide should take parameters (key, Iterator[pandas.DataFrame], state) and return another Iterator[...

  • 0 kudos
Labels