cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Auto Loader: addNewColumns can still stop your stream

karanvichare
New Contributor

A useful detail in the Auto Loader docs: addNewColumns does not mean the stream keeps running when a new column arrives.

Auto Loader updates its saved schema, then stops with UnknownFieldException. Restarting the stream picks up the new schema. Databricks recommends using Lakeflow Jobs to restart automatically.

For production, Iโ€™d also monitor _rescued_data and check that the target Delta table allows the intended schema change. Reader schema evolution and target table evolution need to line up.

Docs: https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema
https://docs.databricks.com/aws/en/data-engineering/schema-evolution

karan
0 REPLIES 0