How to Export lineage data directly from unity catalog without using system tables

[Mahesh_Yadav]

I have been trying to check if there is any direct way to export lineage hierarchy data in data bricks.

I have tried to build a workaround solution by accessing system tables as per this link:

Monitor usage with system tables - Azure Databricks | Microsoft Learn

But want to check if it can be achieved using interface itself instead?

[Kaniz_Fatma]

Hi @Mahesh_Yadav, You can capture lineage data directly from Unity Catalog in Databricks without relying on system tables.

Here’s how you can achieve this:

1. Using Unity Catalog:

   • Go to your Databricks landing page.
   • Click “New” in the sidebar and select “Notebook” from the menu.
   • Enter a name for the notebook and choose “SQL” as the default language.
   • Select a cluster with access to Unity Catalog.
   • Click “Create.”
   • Unity Catalog allows you to capture runtime data lineage across queries run on Databricks. Lineage is supported for all languages and is captured down to the column level. It includes notebooks, workflows, and dashboards related to the query¹.

2. Benefits of Unity Catalog:

   • Unity Catalog provides lineage not only at a table level but also at a row level. This feature is useful for tracking which application is using which data, especially for PII/GDPR d...².

Give this approach a try, and it should help you achieve your goal without relying on system tables! 😊🚀