Could you please help us with this? We are currently blocked in development. When new fields are added to the source JSON, the nested field names are captured in the _rescued_data (rescue) column. However, we do not get any indication of the type of schema operation—such as whether a field was added, deleted, or had its data type changed.
Data type changes and attribute deletions are not captured. In these scenarios, no values are stored in the rescue column. As per the Auto Loader documentation, Auto Loader with rescue data does not capture field deletions or data type changes. Could you please suggest an alternative approach to detect and track these schema changes?