Hi @PushkarDeole, To leverage RocksDB as the state store with `applyInPandasWithState` in Databricks, configure your Spark session with the following setting:
spark.conf.set("spark.sql.streaming.stateStore.providerClass", "com.databricks.sql.streaming.state.RocksDBStateStoreProvider")
Enabling this configuration allows you to manage state using RocksDB for your streaming queries. When using RocksDB, optimize performance by specifying a local directory (`rocksdb.localdir`) for RocksDB's working state, ideally on ephemeral storage like SSDs for faster access. Avoid frequent read/write operations to remote storage such as S3 to prevent performance degradation. Additionally, consider implementing asynchronous checkpoints and utilizing Databricks' state rebalancing features to enhance stateful streaming performance by distributing state evenly across nodes. Explore these options further to optimize your state management setup.
If you have more questions, feel free to ask! ๐