Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-22-2024 03:53 PM
@mvmiller - Per the below documentation, The stream will fail with unknownFieldException, the schema evolution mode by default is addNewColumns. so, Databricks recommends configuring Auto Loader streams with workflows to restart automatically after such schema changes. Incase of interactive cluster workload, can you please restart the cluster to see if the new columns are picked up.
https://learn.microsoft.com/en-us/azure/databricks/ingestion/auto-loader/schema