Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
Is there anyway to accomplish this ? I have an existing Delta Table and a separate Delta Live Table pipelines and I would like to merge data from a DLT to my existing Delta Table.
Seems to me that it is possible. Delta Live tables is in fact a declarative framework to automate a lot in data pipelines. So it is not a table type in fact (I always wondered why they called it like that). As DLT uses Delta Lake, you could create a DLT which merges the other DLTs in your delta lake table, taking into account the limitations of dlt of course.
Merging data from a Delta Live Table (DLT) into an existing Delta Table is possible with careful planning. Transition data from DLT to Delta Table through batch processing, data transformation, and ETL processes, ensuring schema compatibility.
as
Join Us as a Local Community Builder!
Passionate about hosting events and connecting people? Help us grow a vibrant local communityโsign up today to get started!