A useful detail in the Auto Loader docs: addNewColumns does not mean the stream keeps running when a new column arrives.
Auto Loader updates its saved schema, then stops with UnknownFieldException. Restarting the stream picks up the new schema. Databricks recommends using Lakeflow Jobs to restart automatically.
For production, Iโd also monitor _rescued_data and check that the target Delta table allows the intended schema change. Reader schema evolution and target table evolution need to line up.
Docs: https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/schema
https://docs.databricks.com/aws/en/data-engineering/schema-evolution
karan