Questions about Lineage and DLT

[bradleyjamrozik]

Hey there!

1. Does column lineage work across multiple catalogs and schemas?

2. Do Delta Live Tables support lineage? If yes does that work across multiple pipelines or only with a single one?

[erigaud]

Delta live tables support lineage to the table level (as opposed to the column-level lineage provided by Unity Catalog). It capture the lineage for each pipeline individually, so I suppose in a way you could recreate the full lineage yourself.

[Vinay_M_R · Accepted solution]

Hi @bradleyjamrozik 

1. Does column lineage work across multiple catalogs and schemas?

It does state that lineage is aggregated across all workspaces attached to a metastore, meaning that lineage captured in one workspace is visible in any other workspace sharing that metastore. Therefore, it is possible that column lineage could work across multiple catalogs and schemas if they are attached to the same metastore.

2. Do Delta Live Tables support lineage? If yes does that work across multiple pipelines or only with a single one?

Lineage for tables in a Delta Live Tables pipeline is visible in Data Explorer. For materialized views or streaming tables in a Unity Catalog-enabled pipeline, the Data Explorer lineage UI shows the upstream and downstream tables. Lineage is only displayed between tables defined in the pipeline; tables that are defined outside of the pipeline and read in the pipeline are not shown in the Data Explorer lineage UI. 

https://docs.databricks.com/delta-live-tables/unity-catalog.html#view-lineage-for-a-pipeline

Regards,

Vinay M R

[Anonymous]

Hi @bradleyjamrozik 

We haven't heard from you since the last response from @Vinay_M_R and @erigaud , and I was checking back to see if her suggestions helped you.

Or else, If you have any solution, please share it with the community, as it can be helpful to others. 

Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.