cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Governance
Join discussions on data governance practices, compliance, and security within the Databricks Community. Exchange strategies and insights to ensure data integrity and regulatory compliance.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Problems with Unity catalog lineage

andalo
New Contributor II

I have created a catalog, a schema and several tables within unity catalog, when I run a notebook that makes a merge to one of the tables the lineage of this is correct with all its dependencies, but if I run the notebook of another different table the lineage of the previous table is lost, apparently it is as if the lineage information is overwritten with the execution of each notebook. I am using a cluster with runtime 15.4. Can you help me if you can think what could be happening?

1 REPLY 1

Advika
Databricks Employee
Databricks Employee

Hello @andalo!

This might be happening because lineage for MERGE operations is not fully captured by default. Could you try setting the Spark property spark.databricks.dataLineage.mergeIntoV2Enabled to true?
Just a heads-up, enabling this setting may impact query performance slightly.

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local communityโ€”sign up today to get started!

Sign Up Now