szymon_dybczak
Esteemed Contributor III

Hi @yit ,

This is expected behaviour of Auto Loader with schema evolution enabled. Default mode is addNewColumns which causes stream fail. 
As documentation says:

"Auto Loader detects the addition of new columns as it processes your data. When Auto Loader detects a new column, the stream stops with an UnknownFieldException. Before your stream throws this error, Auto Loader performs schema inference on the latest micro-batch of data and updates the schema location with the latest schema by merging new columns to the end of the schema. The data types of existing columns remain unchanged. Databricks recommends configuring Auto Loader streams with Lakeflow Jobs to restart automatically after such schema changes."

And since you didn't specify mode, by deafult addNewColumns will be used:

szymon_dybczak_0-1755000042822.png