Hi @cdn_yyz_yul,

In your new version of the silver stream, you’ve added extra transformations (parsing _rescued_data, join, unpivot). Even though final_silver_df has the same columns as before, this changes the internal schema of at least one stateful operator compared to what’s stored in the checkpoint. As per the structured streaming docs, any change to stateful operations (agg, dedupe, stream‑stream joins, etc.) between restarts from the same checkpoint is unsupported and will fail with a state schema compatibility error. That’s exactly what you’re seeing.

Because you’re on serverless, there’s no supported way to relax this check with Spark configs. The supported migration path is:

1. Deploy the updated silver job (with _rescued_data handling, join, unpivot).
2. Run it with a new checkpointLocation so it rebuilds its state from scratch.
3. Keep the subset of columns that participate in stateful ops stable going forward. Handle future schema drift only in a stateless part of the pipeline (e.g., parse _rescued_data, unpivot, then project back to the same fixed set of stateful columns before any aggregations/joins).

After this one‑time reset, subsequent upstream schema changes in bronze will be absorbed by _rescued_data, and your stateful part won’t need to change, so you won’t hit the state schema error again.

Try this first... print the schema of final_silver_df and compare it to DESCRIBE TABLE silver.my_silver_table. If they match, the error is definitely about the state, not the table schema. You can also temporarily run the updated query with the same write path but a fresh checkpoint dir to confirm it starts and writes successfully. That’s a quick proof that the mismatch is in the checkpointed state.

Hope this helps.