<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:taxo="http://purl.org/rss/1.0/modules/taxonomy/" version="2.0">
  <channel>
    <title>topic Re: Statestore is not valid when using Spark Structured Streaming in Data Engineering</title>
    <link>https://community.databricks.com/t5/data-engineering/statestore-is-not-valid-when-using-spark-structured-streaming/m-p/176469#M57152</link>
    <description>&lt;P&gt;Your screenshot shows the cause. The checkpoint only has metadata, commits and offsets, with no state folder. The state store only exists for stateful operators (aggregations, dropDuplicates, stream-stream joins, windows, transformWithState). A plain readStream → writeStream copy is stateless, so read_statestore has nothing to read and throws STDS_FAILED_TO_READ_STATE_SCHEMA. The “rerun the query” hint doesn’t apply here.&lt;/P&gt;&lt;P&gt;Your audit trail is in the files you do have. metadata holds the query id. offsets/ has one file per batch with the source table version it read up to. commits/ marks which batches finished.&lt;/P&gt;&lt;DIV&gt;&lt;DIV&gt;&lt;DIV&gt;&lt;DIV&gt;&amp;nbsp;&lt;/DIV&gt;&lt;/DIV&gt;&lt;DIV&gt;python&lt;/DIV&gt;&lt;DIV&gt;&lt;PRE&gt;&lt;SPAN&gt;from pyspark.sql import functions as F

cp = "abfss://datalake@abc.dfs.core.windows.net/test002/x05"
query_id = spark.read.json(f"{cp}/metadata").first()["id"]

offsets = (
    spark.read.text(f"{cp}/offsets", wholetext=True)
    .select(
        F.regexp_extract("_metadata.file_path", r"/(\d+)$", 1).cast("long").alias("batch_id"),
        F.get_json_object(F.split("value", "\n").getItem(2), "$.reservoirVersion").cast("long").alias("source_version"),
    )
    .orderBy("batch_id")
)
display(offsets)&lt;/SPAN&gt;&lt;/PRE&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;P&gt;Then filter DESCRIBE HISTORY on your target table to operation = 'STREAMING UPDATE' and operationParameters.queryId = query_id. The epochId parameter is the batch ID, so you can join it to the offsets above to get batch, source version, target version and row count.&lt;/P&gt;&lt;P&gt;One caution: Spark cleans up old offset and commit files (it keeps about the last 100 batches by default), so don’t rely on the checkpoint as a permanent record. For a durable audit trail, write batch_id and row counts to a table from foreachBatch, or persist progress events with a StreamingQueryListener.&lt;/P&gt;</description>
    <pubDate>Mon, 16 Nov 2026 06:06:15 GMT</pubDate>
    <dc:creator>ashishshivatare</dc:creator>
    <dc:date>2026-11-16T06:06:15Z</dc:date>
    <item>
      <title>Statestore is not valid when using Spark Structured Streaming</title>
      <link>https://community.databricks.com/t5/data-engineering/statestore-is-not-valid-when-using-spark-structured-streaming/m-p/176002#M57087</link>
      <description>&lt;P&gt;I'm using spark structured streaming and it works properly at first.&amp;nbsp; But when I try to audit the checkpointLocation I can't get it to work.&amp;nbsp; It seems like it should be a relatively simple task to query the statestore and audit the processing of a stream.&amp;nbsp; I'm attempting to follow these docs:&lt;BR /&gt;&lt;BR /&gt;&lt;A href="https://docs.databricks.com/aws/en/structured-streaming/read-state" target="_blank"&gt;https://docs.databricks.com/aws/en/structured-streaming/read-state&lt;/A&gt;&lt;BR /&gt;&lt;BR /&gt;&lt;SPAN&gt;I'm using dbx 16.4 and spark 3.5.2&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;The error is always something meaningless like so:&lt;/P&gt;&lt;LI-CODE lang="markup"&gt;[STDS_FAILED_TO_READ_STATE_SCHEMA] Failed to read the state schema. Either the file does not exist, or the file is corrupted. options: StateSourceOptions(checkpointLocation=abfss://datalake@xxx.dfs.core.windows.net/test002/x05, batchId=1, operatorId=0, storeName=default, joinSide=none, stateVarName=None, +flattenCollectionTypes=true).

Rerun the streaming query to construct the state schema, and report to the corresponding communities or vendors if the error persists. SQLSTATE: 42K03&lt;/LI-CODE&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;&lt;P&gt;Given the importance of Spark Structured Streaming, and the popularity, I did not think there would be so many sharp edges when auditing the statestore.&lt;/P&gt;&lt;P&gt;My source table is just a UC managed table that is writted by zerobus.&amp;nbsp; It is like so:&lt;/P&gt;&lt;LI-CODE lang="markup"&gt;CREATE TABLE a.b.streaming_c (
  MyA STRING NOT NULL,
  MyB INT NOT NULL,
  MyC STRING,
  MyDetails VARIANT,
  ZerobusRescueJson VARIANT)
USING delta
TBLPROPERTIES (
  'delta.checkpoint.writeStatsAsJson' = 'false',
  'delta.checkpoint.writeStatsAsStruct' = 'true',
  'delta.enableDeletionVectors' = 'true',
  'delta.enableRowTracking' = 'true',
  'delta.feature.appendOnly' = 'supported',
  'delta.feature.deletionVectors' = 'supported',
  'delta.feature.domainMetadata' = 'supported',
  'delta.feature.invariants' = 'supported',
  'delta.feature.rowTracking' = 'supported',
  'delta.feature.variantType' = 'supported',
  'delta.minReaderVersion' = '3',
  'delta.minWriterVersion' = '7',
  'delta.parquet.compression.codec' = 'zstd',
  'delta.parquet.format.version' = '2.12.0',
  'delta.parquet.format.version.afe.internal' = '2.12.0',
  'delta.rowTracking.materializedRowCommitVersionColumnName' = '_row-commit-version-col-777ca68b-c04b-4b5d-9a9d-5c5535e32d20',
  'delta.rowTracking.materializedRowIdColumnName' = '_row-id-col-1885c4cd-1354-4ce6-9726-ed098a251a62',
  'io.unitycatalog.tableId' = '6f07a65b-4887-4aa2-8f56-569b2922aec4')&lt;/LI-CODE&gt;&lt;P&gt;&lt;SPAN&gt;I use &lt;/SPAN&gt;&lt;STRONG&gt;readStream/writeStream&lt;/STRONG&gt;&lt;SPAN&gt; to move the data to a different place. I set the checkpointLocation (ie. .option("&lt;/SPAN&gt;&lt;STRONG&gt;checkpointLocation&lt;/STRONG&gt;&lt;SPAN&gt;", x)) to an abfss location in azure:&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN&gt;"abfss://datalake@abc.dfs.core.windows.net/test002/x05"&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;&lt;P&gt;&lt;span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="DB1To3_0-1794251574660.png" style="width: 400px;"&gt;&lt;img src="https://community.databricks.com/t5/image/serverpage/image-id/33163iC15DD7646A472B60/image-size/medium?v=v2&amp;amp;px=400" role="button" title="DB1To3_0-1794251574660.png" alt="DB1To3_0-1794251574660.png" /&gt;&lt;/span&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Later when the streaming compute stops running, I try to audit the state of the checkpointLocation and it fails:&lt;/SPAN&gt;&lt;/P&gt;&lt;PRE&gt;SELECT * FROM read_statestore('abfss://datalake@abc.dfs.core.windows.net/test002/x05/')&lt;/PRE&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;That &lt;STRONG&gt;checkpointLocation&lt;/STRONG&gt; is very intimidating to me. It is doing ALL the heavy lifting when it comes to keeping track of the zerobus records that have been processed. There needs to be a way to &lt;STRONG&gt;recover an audit trail&lt;/STRONG&gt; from this statestore location. Why is the statestore API giving meaningless errorrs? I tried both the pyspark equivalent of this SELECT statement and it has the same error.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;</description>
      <pubDate>Mon, 09 Nov 2026 19:16:28 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/statestore-is-not-valid-when-using-spark-structured-streaming/m-p/176002#M57087</guid>
      <dc:creator>DB1To3</dc:creator>
      <dc:date>2026-11-09T19:16:28Z</dc:date>
    </item>
    <item>
      <title>Re: Statestore is not valid when using Spark Structured Streaming</title>
      <link>https://community.databricks.com/t5/data-engineering/statestore-is-not-valid-when-using-spark-structured-streaming/m-p/176469#M57152</link>
      <description>&lt;P&gt;Your screenshot shows the cause. The checkpoint only has metadata, commits and offsets, with no state folder. The state store only exists for stateful operators (aggregations, dropDuplicates, stream-stream joins, windows, transformWithState). A plain readStream → writeStream copy is stateless, so read_statestore has nothing to read and throws STDS_FAILED_TO_READ_STATE_SCHEMA. The “rerun the query” hint doesn’t apply here.&lt;/P&gt;&lt;P&gt;Your audit trail is in the files you do have. metadata holds the query id. offsets/ has one file per batch with the source table version it read up to. commits/ marks which batches finished.&lt;/P&gt;&lt;DIV&gt;&lt;DIV&gt;&lt;DIV&gt;&lt;DIV&gt;&amp;nbsp;&lt;/DIV&gt;&lt;/DIV&gt;&lt;DIV&gt;python&lt;/DIV&gt;&lt;DIV&gt;&lt;PRE&gt;&lt;SPAN&gt;from pyspark.sql import functions as F

cp = "abfss://datalake@abc.dfs.core.windows.net/test002/x05"
query_id = spark.read.json(f"{cp}/metadata").first()["id"]

offsets = (
    spark.read.text(f"{cp}/offsets", wholetext=True)
    .select(
        F.regexp_extract("_metadata.file_path", r"/(\d+)$", 1).cast("long").alias("batch_id"),
        F.get_json_object(F.split("value", "\n").getItem(2), "$.reservoirVersion").cast("long").alias("source_version"),
    )
    .orderBy("batch_id")
)
display(offsets)&lt;/SPAN&gt;&lt;/PRE&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;P&gt;Then filter DESCRIBE HISTORY on your target table to operation = 'STREAMING UPDATE' and operationParameters.queryId = query_id. The epochId parameter is the batch ID, so you can join it to the offsets above to get batch, source version, target version and row count.&lt;/P&gt;&lt;P&gt;One caution: Spark cleans up old offset and commit files (it keeps about the last 100 batches by default), so don’t rely on the checkpoint as a permanent record. For a durable audit trail, write batch_id and row counts to a table from foreachBatch, or persist progress events with a StreamingQueryListener.&lt;/P&gt;</description>
      <pubDate>Mon, 16 Nov 2026 06:06:15 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/statestore-is-not-valid-when-using-spark-structured-streaming/m-p/176469#M57152</guid>
      <dc:creator>ashishshivatare</dc:creator>
      <dc:date>2026-11-16T06:06:15Z</dc:date>
    </item>
    <item>
      <title>Re: Statestore is not valid when using Spark Structured Streaming</title>
      <link>https://community.databricks.com/t5/data-engineering/statestore-is-not-valid-when-using-spark-structured-streaming/m-p/176472#M57154</link>
      <description>&lt;P&gt;I think the error is expected for this pipeline. The state reader functions (read_statestore / read_state_metadata) are for stateful Structured Streaming queries, meaning ones with a stateful operator such as aggregations, dropDuplicates, stream-stream joins or arbitrary stateful processing. A plain readStream to writeStream copy doesn't keep any state, so there's no state schema in the checkpoint to read, hence STDS_FAILED_TO_READ_STATE_SCHEMA. You can confirm with read_state_metadata('&amp;lt;checkpoint&amp;gt;'): it should show no stateful operators.&lt;/P&gt;&lt;P&gt;For an audit trail of what was processed, a few options that don't depend on state:&lt;/P&gt;&lt;OL&gt;&lt;LI&gt;Delta history on the target table. Each micro-batch write appears in DESCRIBE HISTORY with its operation metrics (rows written), so you get a per-batch log for free.&lt;/LI&gt;&lt;LI&gt;Add audit columns in the stream itself, for example a processing timestamp with current_timestamp(), or use foreachBatch and write the batchId plus counts to a small audit table alongside the data.&lt;/LI&gt;&lt;LI&gt;Streaming query progress. Each batch's progress (input rows, batch id, source offsets) is available from the query's progress or a StreamingQueryListener, which you can persist to a table.&lt;/LI&gt;&lt;/OL&gt;&lt;P&gt;Also note the state reader has compute requirements (DBR 14.3 LTS+ on dedicated or no-isolation compute, and it isn't supported on serverless), so it's worth checking those if you later add a stateful step.&lt;/P&gt;</description>
      <pubDate>Mon, 16 Nov 2026 06:35:01 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/statestore-is-not-valid-when-using-spark-structured-streaming/m-p/176472#M57154</guid>
      <dc:creator>SajeevPS</dc:creator>
      <dc:date>2026-11-16T06:35:01Z</dc:date>
    </item>
    <item>
      <title>Re: Statestore is not valid when using Spark Structured Streaming</title>
      <link>https://community.databricks.com/t5/data-engineering/statestore-is-not-valid-when-using-spark-structured-streaming/m-p/176527#M57166</link>
      <description>&lt;P&gt;Thanks for the thorough reply.&lt;/P&gt;&lt;P&gt;&lt;STRONG&gt;&amp;gt;&amp;gt;&amp;nbsp; The checkpoint only has metadata, commits and offsets, with no state folder.&lt;/STRONG&gt;&lt;/P&gt;&lt;P&gt;It is a bit counterintuitive to have a bunch of stateful information stored as metadata, commits, and offsets, yet to NOT be able to use a "statestore api" to inspect that information.&lt;/P&gt;&lt;P&gt;&lt;STRONG&gt;&amp;gt;&amp;gt; For a durable audit trail, write batch_id and row counts to a table from foreachBatch, or persist progress events with a StreamingQueryListener.&lt;/STRONG&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Thanks for the reference to "&lt;STRONG&gt;StreamingQueryListener&lt;/STRONG&gt;".&amp;nbsp; That is the sort of turnkey API that I was missing.&amp;nbsp; It wasn't coming up in my searches.&lt;BR /&gt;&lt;BR /&gt;I'm guessing designers of spark streaming did not intend for users to tear apart the &lt;STRONG&gt;checkpointLocation&lt;/STRONG&gt; on our own (although it still seems important to have that ability in a pinch).&amp;nbsp; I was surprised NOT to find a github library or example that creates such audit trails at this level of detail.&amp;nbsp; Maybe with your additional keywords (batch_id, reservoirVersion) then I will find something....&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;FYI, I posted a similar question elsewhere.&amp;nbsp; My underlying goals are not only to be able to audit the checkpointLocation but also to have a strategy for recovering my streams in the event that my checkpointLocation became corrupted one day (for reasons outside my control, for example if I store it on a less dependable location than the place where the deltalake tables live.).&amp;nbsp; I learned about the &lt;STRONG&gt;startingVersion/startingTimestamp&lt;/STRONG&gt; that can be used for recovery.&amp;nbsp; So if I am able to do some checkpointLocation auditing, and make sense of the results, then it will give me better shot at setting startingVersion/startingTimestamp for a point-in-time recovery.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;</description>
      <pubDate>Mon, 16 Nov 2026 14:12:20 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/statestore-is-not-valid-when-using-spark-structured-streaming/m-p/176527#M57166</guid>
      <dc:creator>DB1To3</dc:creator>
      <dc:date>2026-11-16T14:12:20Z</dc:date>
    </item>
    <item>
      <title>Re: Statestore is not valid when using Spark Structured Streaming</title>
      <link>https://community.databricks.com/t5/data-engineering/statestore-is-not-valid-when-using-spark-structured-streaming/m-p/176528#M57167</link>
      <description>&lt;P&gt;Thanks for this reply.&amp;nbsp; I was about to proceed with no. 2, but didn't know about options 1 and 3.&amp;nbsp; I had not come across the StreamingQueryListener yet in my investigation.&amp;nbsp; Perhaps it is only a small fraction of streaming applications that also use that API to monitor the behavior of their solutions.&amp;nbsp; It seems pretty critical for an analysis of failures or misbehaviors.&lt;/P&gt;</description>
      <pubDate>Mon, 16 Nov 2026 14:17:11 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/statestore-is-not-valid-when-using-spark-structured-streaming/m-p/176528#M57167</guid>
      <dc:creator>DB1To3</dc:creator>
      <dc:date>2026-11-16T14:17:11Z</dc:date>
    </item>
  </channel>
</rss>

