Unity Catalog lineage is lost
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-18-2024 02:15 PM
Currently, we have implemented Unity Catalog in Databricks and we are facing an issue with lineage. We execute a notebook that fills a table, and upon completion, the lineage is created correctly; all tables are mapped properly, and we can visualize the lineage tree without any issues. However, the next day, when the notebook is executed automatically as part of the workflow, the lineage information captured the previous day is lost. If we had dependencies on 20 tables, the next day it only shows dependencies for 5 tables.
- Labels:
-
Unity Catalog
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-18-2024 02:17 PM
What does your notebook do? Does it drop the existing tables and create a new one? Give more details about your process preferably the source code.
~
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-18-2024 02:50 PM
The notebook performs a merge between the existing table and new information coming from a dataframe, which is stored in a temporary table.
In the same way, without executing any new operations on the table, the lineage starts to get lost, and tables are removed from the graph
Merge;

