https://community.databricks.com/t5/data-engineering/state-store-configuration-with-applyinpandaswithstate-for/m-p/79093#M35671 <P><a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/9">@Retired_mod</a>&nbsp; thanks for the response. As I understand, there are 3 options to be explored to get optimal performance out of rocksdb based state management:&nbsp;</P><P>1. specify a local directory 'rocksdb.localdir'&nbsp;</P><P>--&gt; will you be able to guide how (through which configuration) this can be specified?&nbsp;</P><P>2. implement asynchronous checkpoints</P><P>--&gt; I looked more into the details of asynchronous checkpoints through this article&nbsp;<A href="https://learn.microsoft.com/en-us/azure/databricks/structured-streaming/async-checkpointing" target="_blank">https://learn.microsoft.com/en-us/azure/databricks/structured-streaming/async-checkpointing</A></P><P>As mentioned in the limitations, cluster resizing might not work well with asynchronous checkpointing. Since we are using auto scaling feature for our databricks cluster, does that mean that we won't be able to use asynchronous checkpointing as it will frequently resize the cluster?</P><P>3. Databricks' state rebalancing</P><P>--&gt; will explore this more</P> Wed, 17 Jul 2024 09:58:50 GMT PushkarDeole 2024-07-17T09:58:50Z https://community.databricks.com/t5/data-engineering/state-store-configuration-with-applyinpandaswithstate-for/m-p/78906#M35634 <P>Hello,</P><P>We are using a stateful pipeline for data processing and analytics. For state store, we are using applyInPandasWithState function however the state needs to be persistent across node restarts etc.&nbsp;</P><P>At this point, we are not sure how the state can be made persistent with applyInPandasWithState. There are some articles where it is mentioned around usage of RocksDB state store for persistence</P><P>Couple of questions:</P><P>1. What configurations is required to enable RocksDB state storage with applyInPandasWithState ?</P><P>2. What are the tuning parameters for RocksDB state store that can be tuned to provide optimal performance?</P><P>Any guidance around these would be appreciated.&nbsp;</P><P>&nbsp;</P> Tue, 16 Jul 2024 05:20:43 GMT https://community.databricks.com/t5/data-engineering/state-store-configuration-with-applyinpandaswithstate-for/m-p/78906#M35634 PushkarDeole 2024-07-16T05:20:43Z https://community.databricks.com/t5/data-engineering/state-store-configuration-with-applyinpandaswithstate-for/m-p/79093#M35671 <P><a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/9">@Retired_mod</a>&nbsp; thanks for the response. As I understand, there are 3 options to be explored to get optimal performance out of rocksdb based state management:&nbsp;</P><P>1. specify a local directory 'rocksdb.localdir'&nbsp;</P><P>--&gt; will you be able to guide how (through which configuration) this can be specified?&nbsp;</P><P>2. implement asynchronous checkpoints</P><P>--&gt; I looked more into the details of asynchronous checkpoints through this article&nbsp;<A href="https://learn.microsoft.com/en-us/azure/databricks/structured-streaming/async-checkpointing" target="_blank">https://learn.microsoft.com/en-us/azure/databricks/structured-streaming/async-checkpointing</A></P><P>As mentioned in the limitations, cluster resizing might not work well with asynchronous checkpointing. Since we are using auto scaling feature for our databricks cluster, does that mean that we won't be able to use asynchronous checkpointing as it will frequently resize the cluster?</P><P>3. Databricks' state rebalancing</P><P>--&gt; will explore this more</P> Wed, 17 Jul 2024 09:58:50 GMT https://community.databricks.com/t5/data-engineering/state-store-configuration-with-applyinpandaswithstate-for/m-p/79093#M35671 PushkarDeole 2024-07-17T09:58:50Z