Problems with Unity catalog lineage
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-21-2025 02:24 PM - edited 05-21-2025 02:25 PM
I have created a catalog, a schema and several tables within unity catalog, when I run a notebook that makes a merge to one of the tables the lineage of this is correct with all its dependencies, but if I run the notebook of another different table the lineage of the previous table is lost, apparently it is as if the lineage information is overwritten with the execution of each notebook. I am using a cluster with runtime 15.4. Can you help me if you can think what could be happening?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-22-2025 03:24 AM
Hello @andalo!
This might be happening because lineage for MERGE operations is not fully captured by default. Could you try setting the Spark property spark.databricks.dataLineage.mergeIntoV2Enabled to true?
Just a heads-up, enabling this setting may impact query performance slightly.