Merge DLT with Delta Table
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-12-2023 10:57 PM
Is there anyway to accomplish this ? I have an existing Delta Table and a separate Delta Live Table pipelines and I would like to merge data from a DLT to my existing Delta Table.
Is this doable or completely impossible ?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-13-2023 12:28 AM
Seems to me that it is possible. Delta Live tables is in fact a declarative framework to automate a lot in data pipelines.
So it is not a table type in fact (I always wondered why they called it like that). As DLT uses Delta Lake, you could create a DLT which merges the other DLTs in your delta lake table, taking into account the limitations of dlt of course.
Something like this
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-14-2023 02:43 AM
This is doable, use LIVE keyword while matching the delta table to delta live table in merge.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-03-2023 11:06 PM
Merging data from a Delta Live Table (DLT) into an existing Delta Table is possible with careful planning. Transition data from DLT to Delta Table through batch processing, data transformation, and ETL processes, ensuring schema compatibility.
