Hi @cdn_yyz_yul,
Because the silver stream runs on serverless, you can’t relax state-store schema checks or set custom Spark configs. When the upstream bronze table schema evolves in a way that changes the schema of any stateful operator, the streaming query will correctly fail with STATE_STORE_VALUE_SCHEMA_NOT_COMPATIBLE.
On serverless, the supported pattern is to treat this as a breaking change:
- Stop the silver streaming job.
- Update your code/schema for the new upstream schema.
- Restart the stream with a new checkpoint location (or delete/rotate the existing checkpoint) so that the state is rebuilt with the new schema.
This is expected behaviour. The documentation here lists the Spark properties you can configure on Serverless. It also says the below...
You can also refer to this page, which lists Serverless's limitations. It'll again take you to the same page mentioned above.
If you want to avoid rebuilding state on every breaking schema change, you can run the silver stream on classic/job compute instead of serverless, where you can control Spark configs and design more flexible state-handling. However, I appreciate that it may not be what you are looking for.
If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.
