Hi @cdn_yyz_yul,

Great. Both of the approaches you’re testing are reasonable ways to protect your stateful silver stream from upstream schema changes on serverless...

On the first option, with schemaEvolutionMode = "rescue" on CSV... you can keep Auto Loader in CSV but switch to.. 

.option("cloudFiles.schemaEvolutionMode", "rescue")
.option("rescuedDataColumn", "_rescued_data")

In silver, you first do stateless parsing/flattening of _rescued_data into whatever extra fields you need and feed only a stable, fixed projection of columns into the stateful part of the query (aggregations/joins, etc.).

As long as the columns that participate in stateful operations don’t change schema, the state-store schema stays compatible, and you avoid STATE_STORE_*_SCHEMA_NOT_COMPATIBLE without needing any forbidden Spark confs on serverless.

The other option, where you are talking about keeping the format as text and parsing at silver, this is a more extreme version of the same idea. It does give you maximum isolation from source changes, but you lose CSV parsing at bronze and push more work into silver. You’d still want the same pattern... as in parse in a stateless layer, then project into a stable set of columns before any stateful operators.

Something to remember here is that even with these patterns, any intentional change to the schema of your stateful part in Silver (for example, adding new grouping keys or changing types used in aggregations/joins) is still a breaking state change and requires starting that stream with a new checkpoint location. That’s independent of serverless. It’s how Structured Streaming state recovery works.

Hope this helps.