Databricks

FreshPork · ‎10-02-2023

Hi,

Is it possible to track and display data lineage with Unity Catalog while using databricks on GCP?
like so: https://docs.databricks.com/en/data-governance/unity-catalog/data-lineage.html

If it's not yet implemented, then is there any roadmap available, which would give us an indication of when this functionality might be enabled on GCP?

Thanks for help!

Kaniz · ‎10-05-2023

Hi @FreshPork, Yes, it is possible to track and display data lineage with Unity Catalog while using Databricks on GCP.

You can use the Unity Catalog to capture runtime data lineage across queries on Databricks. Lineage is supported for all languages and is captured down to the column level. Lineage data includes notebooks, workflows, and dashboards related to the query. Lineage can be visualized in Data Explorer in near real-time and retrieved with the Databricks REST API.

View solution in original post

Kaniz · ‎10-05-2023

Hi @FreshPork, Yes, it is possible to track and display data lineage with Unity Catalog while using Databricks on GCP.

You can use the Unity Catalog to capture runtime data lineage across queries on Databricks. Lineage is supported for all languages and is captured down to the column level. Lineage data includes notebooks, workflows, and dashboards related to the query. Lineage can be visualized in Data Explorer in near real-time and retrieved with the Databricks REST API.