topic Re: Data Lineage with Apply Changes in Data Engineering

Data Lineage with Apply Changes

Gareema — Sun, 07 Jul 2024 19:20:03 GMT

Hello Team

I am using DLT. I am able to see the lineage when doing normal process. However as soon as I use 'APPLY_Changes' feature after the lineag ebreaks and I am no more able to see the Data lineage from the catalog after going to table.

Is there any way that I can use Apply_Changes and have the lineage retained?

Code to load in silver table:

Re: Data Lineage with Apply Changes

Gareema — Sun, 07 Jul 2024 19:35:04 GMT

One point here is, both the tables are in different schema. Hence different DLT pipelines and notebooks are there.

Re: Data Lineage with Apply Changes

Gareema — Mon, 08 Jul 2024 14:32:08 GMT

@Retired_mod Thank you for your response. However, in both approaches we are avoiding the 'APPLY_Changes' feature. In reality I want to utilise that feature to have the latest record based on the 'Sequence Key'.

With merge or overwrite I will not be able to have the flexibility of getting just the last updated row based on sequence.

Re: Data Lineage with Apply Changes

Gareema — Mon, 08 Jul 2024 16:20:21 GMT

@Retired_mod : Is there any way this can be achieved or can we expect this problem to be resolved in next releases?