Handling New Columns Using Auto Loader Rescue Mode but how will get newly added column

[SantiNath_Dey]

Schema evolution (rescue mode) is being triggered for a complex JSON file. However, when a new column is added in the source, it does not appear in the target table automatically for multi-level nested JSON. How can we add newly added nested columns to the target table when using Databricks Auto Loader with rescue mode?

[Ashwin_DSA]

Hi @SantiNath_Dey,

In rescue mode, Auto Loader intentionally freezes the schema. This is by design. New columns, whether top-level or nested, will not appear in the target, they land in _rescued_data as a JSON string. That is the mode's contract. 

Refer to this page.

You'll have to switch the evolution mode to resolve this...

While you can consider other options like pre-declaring the fields you need using something like schemahints, this only works when you know which fields you are sure about. It's not something you'd want to use for complex json structures. Rescue mode is best thought of as a safety net for unexpected data, not as an evolution strategy. 

Is there a specific reason why you'd want to do that?

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.