I'm encountering the following error while trying to upload a RocksDB checkpoint in Databricks:

java.lang.IllegalStateException: Found no SST files during uploading RocksDB checkpoint version 498 with 2332 key(s).
    at com.databricks.sql.streaming.state.RocksDBFileManager.verifyImmutableFiles(RocksDBFileManager.scala:620)
    at com.databricks.sql.streaming.state.RocksDBFileManager.saveCheckpointToDbfs(RocksDBFileManager.scala:173)
    at com.databricks.sql.rocksdb.CloudRocksDB.$anonfun$sync$7(CloudRocksDB.scala:235)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
    at org.apache.spark.util.Utils$.timeTakenMs(Utils.scala:668)
    at com.databricks.sql.rocksdb.CloudRocksDB.timeTakenMs(CloudRocksDB.scala:634)
    at com.databricks.sql.rocksdb.CloudRocksDB.$anonfun$sync$1(CloudRocksDB.scala:234)
    at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)
    at com.databricks.logging.UsageLogging.$anonfun$recordOperation$1(UsageLogging.scala:395)

Background:

I am using Databricks with Spark and RocksDB for stateful streaming. This error occurs when Spark attempts to upload a RocksDB checkpoint, and the system reports that no SST files were found.

• What could be causing this error and why are no SST files being found during the upload process?

• Are there any specific configurations or setups I might be missing for properly handling RocksDB checkpoints in Databricks?

• What potential solutions or workarounds exist for this issue?

We are using all default Spark runtime configurations.

SPARK Version: 2.x

Current Work-around:

Checkpoint needs to be deleted from s3 and retrigger of the streaming pipeline is fixed