topic Problems with Unity catalog lineage in Data Governance https://community.databricks.com/t5/data-governance/problems-with-unity-catalog-lineage/m-p/119920#M2487 <P>I have created a catalog, a schema and several tables within unity catalog, when I run a notebook that makes a merge to one of the tables the lineage of this is correct with all its dependencies, but if I run the notebook of another different table the lineage of the previous table is lost, apparently it is as if the lineage information is overwritten with the execution of each notebook. I am using a cluster with runtime 15.4. Can you help me if you can think what could be happening?</P> Wed, 21 May 2025 21:25:44 GMT andalo 2025-05-21T21:25:44Z Problems with Unity catalog lineage https://community.databricks.com/t5/data-governance/problems-with-unity-catalog-lineage/m-p/119920#M2487 <P>I have created a catalog, a schema and several tables within unity catalog, when I run a notebook that makes a merge to one of the tables the lineage of this is correct with all its dependencies, but if I run the notebook of another different table the lineage of the previous table is lost, apparently it is as if the lineage information is overwritten with the execution of each notebook. I am using a cluster with runtime 15.4. Can you help me if you can think what could be happening?</P> Wed, 21 May 2025 21:25:44 GMT https://community.databricks.com/t5/data-governance/problems-with-unity-catalog-lineage/m-p/119920#M2487 andalo 2025-05-21T21:25:44Z Re: Problems with Unity catalog lineage https://community.databricks.com/t5/data-governance/problems-with-unity-catalog-lineage/m-p/119949#M2488 <P>Hello&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/78439">@andalo</a>!</P> <P>This might be happening because lineage for MERGE operations is not fully captured by default. Could you try setting the Spark property <EM>spark.databricks.dataLineage.mergeIntoV2Enabled</EM> to true?<BR />Just a heads-up, enabling this setting may impact query performance slightly.</P> Thu, 22 May 2025 10:24:02 GMT https://community.databricks.com/t5/data-governance/problems-with-unity-catalog-lineage/m-p/119949#M2488 Advika 2025-05-22T10:24:02Z