topic Re: Questions about Lineage and DLT in Data Engineering

Questions about Lineage and DLT

bradleyjamrozik — Thu, 20 Jul 2023 20:03:58 GMT

Hey there!

1. Does column lineage work across multiple catalogs and schemas?

2. Do Delta Live Tables support lineage? If yes does that work across multiple pipelines or only with a single one?

Re: Questions about Lineage and DLT

erigaud — Fri, 21 Jul 2023 06:47:56 GMT

Delta live tables support lineage to the table level (as opposed to the column-level lineage provided by Unity Catalog). It capture the lineage for each pipeline individually, so I suppose in a way you could recreate the full lineage yourself.

Re: Questions about Lineage and DLT

Vinay_M_R — Fri, 21 Jul 2023 07:07:09 GMT

Hi @bradleyjamrozik

1. Does column lineage work across multiple catalogs and schemas?

It does state that lineage is aggregated across all workspaces attached to a metastore, meaning that lineage captured in one workspace is visible in any other workspace sharing that metastore. Therefore, it is possible that column lineage could work across multiple catalogs and schemas if they are attached to the same metastore.

2. Do Delta Live Tables support lineage? If yes does that work across multiple pipelines or only with a single one?

Lineage for tables in a Delta Live Tables pipeline is visible in Data Explorer. For materialized views or streaming tables in a Unity Catalog-enabled pipeline, the Data Explorer lineage UI shows the upstream and downstream tables. Lineage is only displayed between tables defined in the pipeline; tables that are defined outside of the pipeline and read in the pipeline are not shown in the Data Explorer lineage UI.

https://docs.databricks.com/delta-live-tables/unity-catalog.html#view-lineage-for-a-pipeline

Regards,

Vinay M R

Re: Questions about Lineage and DLT

Anonymous — Sun, 23 Jul 2023 04:39:25 GMT

Hi @bradleyjamrozik

We haven't heard from you since the last response from @Vinay_M_R and @erigaud , and I was checking back to see if her suggestions helped you.

Or else, If you have any solution, please share it with the community, as it can be helpful to others.

Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.