RocksDB results in empty stream/stream joins dataframe

EhsanSaba
New Contributor

‎04-24-2024 02:32 AM

Since we enable RocksDB in our spark.conf the stream to stream joins/unions results in empty dataframe, does anyone else have the same experience? it is on AWS

spark.conf.set(
"spark.sql.streaming.stateStore.providerClass",
"com.databricks.sql.streaming.state.RocksDBStateStoreProvider")

0 Kudos

jose_gonzalez
Databricks Employee
Databricks Employee

‎04-30-2024 03:55 PM

Did you also update the checkpoint? You might need to use a new checkpoint after you enable the RocksDB state store.

0 Kudos