Data Lineage with Apply Changes

Gareema
Contributor

‎07-07-2024 12:20 PM

Hello Team

I am using DLT. I am able to see the lineage when doing normal process. However as soon as I use 'APPLY_Changes' feature after the lineag ebreaks and I am no more able to see the Data lineage from the catalog after going to table. 

Is there any way that I can use Apply_Changes and have the lineage retained? 

Code to load in silver table:

Gareema_0-1720379993368.png

 

0 Kudos

Gareema
Contributor

‎07-07-2024 12:35 PM

One point here is, both the tables are in different schema. Hence different DLT pipelines and notebooks are there.

Gareema
Contributor

‎07-08-2024 07:32 AM

@Retired_mod Thank you for your response. However, in both approaches we are avoiding the 'APPLY_Changes' feature. In reality I want to utilise that feature to have the latest record based on the 'Sequence Key'. 

With merge or overwrite I will not be able to have the flexibility of getting just the last updated row based on sequence. 

Gareema
Contributor

‎07-08-2024 09:20 AM

@Retired_mod : Is there any way this can be achieved or can we expect this problem to be resolved in next releases?

0 Kudos