Need help with DLT
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-18-2024 03:36 AM
I have a DLT pipeline on databricks that has been running since months and just now I found out that there has been an issue with a logic in silver layer and as a result, the tables in my silver schema has faulty records now. Silver layer tables are STREAMING TABLES.
I want to fix this issue in DLT without creating new set of tables. So, what I need to do is to read from my streaming tables in silver, do my logic update, delete the faulty records from the same streaming table in silver and then write the correct records back to the same silver layer streaming schema.(In other words: Updating silver schema)
Is it possible to do it this way i.e. reading from, updating and writing to the same schema in the context of DLT?
Any quick help would be highly appreciated.