topic DLT Pipeline Design in Get Started Discussions

DLT Pipeline Design

Anubhav2603 — Thu, 21 Aug 2025 05:56:02 GMT

I am new to DLT and trying to understand the process. My bronze table will receive incremental data from SAP in real time. In my bronze table, we are not maintaining any history and any data older than 2 weeks will be deleted. This data from bronze will move to my Silver table incrementally. My silver table will have all the history. The mapping between bronze and silver is direct 1 to 1 with no transformation. We are just enforcing schema and cleaning the data little bit. My bronze table is append only. But my silver table will have upsert operation with scd type 1. I also need to have a change_timestamp and an operation_flag column in my silver table. How to do this in DLT?

Re: DLT Pipeline Design

BR_DatabricksAI — Thu, 21 Aug 2025 07:02:33 GMT

Hello @Anubhav2603 : Scenario which you have highlighted can be easily handled using dlt table along with cdc flow. If i understood the scenario bronze table is always append. Have a look the below link. hope this help.

CDC pipeline with Delta Live Table.

Re: DLT Pipeline Design

alexajames — Wed, 27 Aug 2025 14:52:58 GMT

The scenario you mentioned can be efficiently managed using a Delta Live Table (DLT) combined with a CDC (Change Data Capture) flow. From what I understand, the bronze table is always in append mode. Check out the link below for more details. Hope this helps!

CDC pipeline with Delta Live Table .