Hi @cdn_yyz_yul,

Your experiment confirms the key point that with only _rescued_data handling + unpivot, the silver stream is effectively stateless, so reusing the same checkpoint works fine, even as bronze evolves. As soon as you add the join, you introduce state, and when the upstream change affects the join’s input schema, the state-store schema no longer matches what’s stored under the existing checkpoint. The restart from that checkpoint then correctly fails with a state schema mismatch, even though the final DataFrame/table schema looks unchanged.

On serverless, you also can’t bypass these checks using Spark configurations (no spark.sql.streaming.stateStore.*, no delta auto‑merge configs), so there isn’t a supported way to automatically evolve the state while keeping the same checkpoint. I think what you’re doing today is therefore the pragmatic, supported pattern...

And to your specific question about using _rescued_data vs. addNewColumns, they are equivalent for the checkpoint behaviour in silver layer. The reasons you might still choose one over the other are table‑design/downstream concerns, not streaming‑state concerns.

From my perspective, addNewColumns approach means simpler code, no JSON parsing. Howeverm your bronze (and possibly silver) schema keeps growing wider as every new field becomes a real column.

If you use rescue + _rescued_data, it keeps the physical schema more stable and captures unexpected fields and type mismatches into one semi‑structured column. But you pay with extra parsing logic if you want to promote some of those rescued fields.

So if your main goal is to minimise implementation complexity, and you’re already accepting that a new silver checkpoint is needed on those breaking changes, then preferring schemaEvolutionMode = "addNewColumns" is a perfectly reasonable and simpler choice.

Hope this helps.