DLT Pipeline Question

[Anubhav2603]

I am new to DLT and trying to understand the process. My bronze table will receive incremental data from SAP in real time. In my bronze table, we are not maintaining any history and any data older than 2 weeks will be deleted. This data from bronze will move to my Silver table incrementally. My silver table will have all the history. The mapping between bronze and silver is direct 1 to 1 with no transformation. We are just enforcing schema and cleaning the data little bit. My bronze table is append only. But my silver table will have upsert operation with scd type 1. I also need to have a change_timestamp and an operation_flag column in my silver table. How to do this in DLT?

[BigRoux]

When loading data from SAP, how will you determine which records are new? With Lakeflow, incremental loads from cloud storage or another Delta table are fully automated. However, when pulling directly from SAP, Lakeflow does not have visibility into which records were loaded during the previous run.