topic Re: Merge DLT with Delta Table in Data Engineering

Merge DLT with Delta Table

erigaud — Fri, 13 Oct 2023 05:57:03 GMT

Is there anyway to accomplish this ? I have an existing Delta Table and a separate Delta Live Table pipelines and I would like to merge data from a DLT to my existing Delta Table.

Is this doable or completely impossible ?

Re: Merge DLT with Delta Table

-werners- — Fri, 13 Oct 2023 07:28:38 GMT

Seems to me that it is possible. Delta Live tables is in fact a declarative framework to automate a lot in data pipelines.
So it is not a table type in fact (I always wondered why they called it like that). As DLT uses Delta Lake, you could create a DLT which merges the other DLTs in your delta lake table, taking into account the limitations of dlt of course.

Something like this

Re: Merge DLT with Delta Table

Faisal — Sat, 14 Oct 2023 09:43:03 GMT

This is doable, use LIVE keyword while matching the delta table to delta live table in merge.

Re: Merge DLT with Delta Table

LeifBruen — Sat, 04 Nov 2023 06:06:22 GMT

Merging data from a Delta Live Table (DLT) into an existing Delta Table is possible with careful planning. Transition data from DLT to Delta Table through batch processing, data transformation, and ETL processes, ensuring schema compatibility.