Databricks UC enabled but Lineage not found for one table
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-05-2024 09:59 PM
Databricks UC enabled but Lineage not found for one table whereas i can see the lineage for the other two, any idea on why is it?. Im performing few transformations to bronze data , taking good_data_transformed as a dataframe, creating a temp view for that, using that temp view, i create a source_df and merge into silver table.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-06-2024 02:18 AM
Is this from the UI or from the lineage table?
Probably worth checking both. The database option is in system catalog, so system.access.table_lineage
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-06-2024 04:12 AM
i cant see in lineage in UI
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-06-2024 04:30 AM
Like I said, worth checking the database too.....
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-06-2024 04:46 AM
its not there too
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-06-2024 04:48 AM
Right now,Im using good_transformed_df directly to merge with silver
good_transformed_df- which is output of transformation func to which bronze data is feb. But still its not showing up.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-06-2024 07:47 AM
It is because of using temp view. To debug further you would need to write all the source tables, transformations, target tables, actual lineage and expected lineage, but as a rule of thumb if the lineage is lost when using temp view.
Lineage is captured at the table level, and it's most reliably tracked when both the source and target are referenced by table names. If you're creating a temporary view and then using that view to create a source_df which is then merged into a silver table, it's possible that the lineage tracking system is unable to capture this lineage due to the use of temporary view.