Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-14-2024 01:08 AM
You can do this using Delta Live Tables, For design you can follow medallion architecture (https://www.databricks.com/glossary/medallion-architecture).
You can have your 1st table as bronze with everything just appended and second table as silver with defined keys and selected columns.
You can take a look here for how to implement this - https://docs.databricks.com/en/delta-live-tables/tutorial-pipelines.html
Pratik Jain