Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-11-2026 10:58 PM
We are implementing an incremental load for semi-structured data [ complex nested Json file ]using Auto Loader. To handle schema drifts—such as new fields, changes in column order, or data type and precision modifications (e.g., Decimal and Integer)—we are utilizing 'rescue' mode. If these changes occur in a subsequent batch, the pipeline will log the mismatched entries into a Delta table and trigger a Databricks Workflow email notification containing a full description of the drift. Could you please suggest the approach and sudo code. Thanks in advance.