โ09-05-2024 09:59 PM
Databricks UC enabled but Lineage not found for one table whereas i can see the lineage for the other two, any idea on why is it?. Im performing few transformations to bronze data , taking good_data_transformed as a dataframe, creating a temp view for that, using that temp view, i create a source_df and merge into silver table.
โ09-06-2024 02:18 AM
Is this from the UI or from the lineage table?
Probably worth checking both. The database option is in system catalog, so system.access.table_lineage
โ09-06-2024 04:12 AM
i cant see in lineage in UI
โ09-06-2024 04:30 AM
Like I said, worth checking the database too.....
โ09-06-2024 04:46 AM
its not there too
โ09-06-2024 04:48 AM
Right now,Im using good_transformed_df directly to merge with silver
good_transformed_df- which is output of transformation func to which bronze data is feb. But still its not showing up.
โ09-06-2024 07:47 AM
It is because of using temp view. To debug further you would need to write all the source tables, transformations, target tables, actual lineage and expected lineage, but as a rule of thumb if the lineage is lost when using temp view.
Lineage is captured at the table level, and it's most reliably tracked when both the source and target are referenced by table names. If you're creating a temporary view and then using that view to create a source_df which is then merged into a silver table, it's possible that the lineage tracking system is unable to capture this lineage due to the use of temporary view.
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group