topic Unity Catalog lineage is lost in Data Governance

Unity Catalog lineage is lost

andalo — Fri, 18 Oct 2024 21:15:24 GMT

Currently, we have implemented Unity Catalog in Databricks and we are facing an issue with lineage. We execute a notebook that fills a table, and upon completion, the lineage is created correctly; all tables are mapped properly, and we can visualize the lineage tree without any issues. However, the next day, when the notebook is executed automatically as part of the workflow, the lineage information captured the previous day is lost. If we had dependencies on 20 tables, the next day it only shows dependencies for 5 tables.

Re: Unity Catalog lineage is lost

gchandra — Fri, 18 Oct 2024 21:17:06 GMT

What does your notebook do? Does it drop the existing tables and create a new one? Give more details about your process preferably the source code.

Re: Unity Catalog lineage is lost

andalo — Fri, 18 Oct 2024 21:50:34 GMT

The notebook performs a merge between the existing table and new information coming from a dataframe, which is stored in a temporary table.

In the same way, without executing any new operations on the table, the lineage starts to get lost, and tables are removed from the graph

Merge;

deltaTable.alias("data").merge(

dfUpdates.alias("newData"),

condition

).whenMatchedUpdate(set=update_columns) \

.whenNotMatchedInsert(values=dic_columns) \

.whenNotMatchedBySourceUpdate(condition="data.eliminado = 0 " ,set={"data.eliminado": "1"}) \

.execute()