Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-16-2026 04:27 PM
The first option, setting the bronze table to use rescue.
In silver layer,
- - read the table from bronze
- processed the json in _rescued_data
- added rescued columns as new columns to the dataframe in silver, e.g., silver_df1.
- I do need to join (silver_df1 with another existing df) to get "enriched_df1"
- then, did a stack to "unpivot" the enriched_df1, turning some columns, including the rescued columns into rows. gets final_silver_df. (the unpivot method is similar to
- At this stage, the schema of the final_silver_df is the same as before, but writing to the silver delta table (which is created before columns are rescued in bronze) still fails with schema mismatch.
