Unity Catalog lineage is lost

[andalo]

Currently, we have implemented Unity Catalog in Databricks and we are facing an issue with lineage. We execute a notebook that fills a table, and upon completion, the lineage is created correctly; all tables are mapped properly, and we can visualize the lineage tree without any issues. However, the next day, when the notebook is executed automatically as part of the workflow, the lineage information captured the previous day is lost. If we had dependencies on 20 tables, the next day it only shows dependencies for 5 tables.

[gchandra]

What does your notebook do? Does it drop the existing tables and create a new one? Give more details about your process preferably the source code.

[andalo]

The notebook performs a merge between the existing table and new information coming from a dataframe, which is stored in a temporary table.

In the same way, without executing any new operations on the table, the lineage starts to get lost, and tables are removed from the graph

Merge;

deltaTable.alias("data").merge(

            dfUpdates.alias("newData"),

            condition

        ).whenMatchedUpdate(set=update_columns) \

         .whenNotMatchedInsert(values=dic_columns) \

         .whenNotMatchedBySourceUpdate(condition="data.eliminado = 0 " ,set={"data.eliminado": "1"}) \

         .execute()