schema evolution with structured streaming: upstream schema change causes downstream writer fails.

cdn_yyz_yul
Contributor III

Hello,

Bronze: use classic or job compute,  Autoloader with.option("mergeSchema", "true"). Schema evolution works correctly. data goes to bronze.my_bronze_table.

Silver: uses serverless compute, reader reads bronze.my_bronze_table, does all necessary transformation, writer creates silver.my_silver_table, which has defined schema. 

The problem I am trying to resolve:
whenever the schema of bronze.my_bronze_table is changed (thanks to schema evolution), the writer fails to write silver.my_silver_table due to [STATE_STORE_VALUE_SCHEMA_NOT_COMPATIBLE] The provided value schema does not match existing schema in operator state. 

The detailed debug log says clearly that the mismatch is in the schema of bronze.my_bronze_table. 

As the silver uses serverless compute, I can not set 

"spark.databricks.delta.schema.autoMerge.enabled", 
.option("mergeSchema", "true") by itself does not work for the silver table writer. 
 
The current workaround is to delete the checkpoint of silver.my_silver_table. 
 
Is there a better solution to this problem?
 
Thank you!