Hi @Mahesh_Yadav, You can capture lineage data directly from Unity Catalog in Databricks without relying on system tables.
Here’s how you can achieve this:
-
Using Unity Catalog:
- Go to your Databricks landing page.
- Click “New” in the sidebar and select “Notebook” from the menu.
- Enter a name for the notebook and choose “SQL” as the default language.
- Select a cluster with access to Unity Catalog.
- Click “Create.”
- Unity Catalog allows you to capture runtime data lineage across queries run on Databricks. Lineage is supported for all languages and is captured down to the column level. It includes notebooks, workflows, and dashboards related to the query1.
-
Benefits of Unity Catalog:
Give this approach a try, and it should help you achieve your goal without relying on system tables! 😊🚀