Workflow Notification: Pass/Failed –Schema Evolution/Rescue Mode Triggered for complex json file

SantiNath_Dey
Contributor

We are implementing an incremental load for semi-structured data [ complex nested Json file ]using Auto Loader. To handle schema drifts—such as new fields, changes in column order, or data type and precision modifications (e.g., Decimal and Integer)—we are utilizing 'rescue' mode. If these changes occur in a subsequent batch, the pipeline will log the mismatched entries into a Delta table and trigger a Databricks Workflow email notification containing a full description of the drift. Could you please suggest the approach and sudo code. Thanks in advance.