Statestore is not valid when using Spark Structured Streaming

DB1To3 — Mon, 09 Nov 2026 19:16:28 GMT

I'm using spark structured streaming and it works properly at first. But when I try to audit the checkpointLocation I can't get it to work. It seems like it should be a relatively simple task to query the statestore and audit the processing of a stream. I'm attempting to follow these docs:

https://docs.databricks.com/aws/en/structured-streaming/read-state

I'm using dbx 16.4 and spark 3.5.2

The error is always something meaningless like so:

[STDS_FAILED_TO_READ_STATE_SCHEMA] Failed to read the state schema. Either the file does not exist, or the file is corrupted. options: StateSourceOptions(checkpointLocation=abfss://datalake@xxx.dfs.core.windows.net/test002/x05, batchId=1, operatorId=0, storeName=default, joinSide=none, stateVarName=None, +flattenCollectionTypes=true). Rerun the streaming query to construct the state schema, and report to the corresponding communities or vendors if the error persists. SQLSTATE: 42K03

Given the importance of Spark Structured Streaming, and the popularity, I did not think there would be so many sharp edges when auditing the statestore.

My source table is just a UC managed table that is writted by zerobus. It is like so:

CREATE TABLE a.b.streaming_c ( MyA STRING NOT NULL, MyB INT NOT NULL, MyC STRING, MyDetails VARIANT, ZerobusRescueJson VARIANT) USING delta TBLPROPERTIES ( 'delta.checkpoint.writeStatsAsJson' = 'false', 'delta.checkpoint.writeStatsAsStruct' = 'true', 'delta.enableDeletionVectors' = 'true', 'delta.enableRowTracking' = 'true', 'delta.feature.appendOnly' = 'supported', 'delta.feature.deletionVectors' = 'supported', 'delta.feature.domainMetadata' = 'supported', 'delta.feature.invariants' = 'supported', 'delta.feature.rowTracking' = 'supported', 'delta.feature.variantType' = 'supported', 'delta.minReaderVersion' = '3', 'delta.minWriterVersion' = '7', 'delta.parquet.compression.codec' = 'zstd', 'delta.parquet.format.version' = '2.12.0', 'delta.parquet.format.version.afe.internal' = '2.12.0', 'delta.rowTracking.materializedRowCommitVersionColumnName' = '_row-commit-version-col-777ca68b-c04b-4b5d-9a9d-5c5535e32d20', 'delta.rowTracking.materializedRowIdColumnName' = '_row-id-col-1885c4cd-1354-4ce6-9726-ed098a251a62', 'io.unitycatalog.tableId' = '6f07a65b-4887-4aa2-8f56-569b2922aec4')

I use readStream/writeStream to move the data to a different place. I set the checkpointLocation (ie. .option("checkpointLocation", x)) to an abfss location in azure:
"abfss://datalake@abc.dfs.core.windows.net/test002/x05"

Later when the streaming compute stops running, I try to audit the state of the checkpointLocation and it fails:

SELECT * FROM read_statestore('abfss://datalake@abc.dfs.core.windows.net/test002/x05/')

That checkpointLocation is very intimidating to me. It is doing ALL the heavy lifting when it comes to keeping track of the zerobus records that have been processed. There needs to be a way to recover an audit trail from this statestore location. Why is the statestore API giving meaningless errorrs? I tried both the pyspark equivalent of this SELECT statement and it has the same error.

topic Statestore is not valid when using Spark Structured Streaming in Data Engineering

Statestore is not valid when using Spark Structured Streaming