2 weeks ago
Databricks UC enabled but Lineage not found for one table whereas i can see the lineage for the other two, any idea on why is it?. Im performing few transformations to bronze data , taking good_data_transformed as a dataframe, creating a temp view for that, using that temp view, i create a source_df and merge into silver table.
2 weeks ago
Is this from the UI or from the lineage table?
Probably worth checking both. The database option is in system catalog, so system.access.table_lineage
2 weeks ago
i cant see in lineage in UI
2 weeks ago
Like I said, worth checking the database too.....
2 weeks ago
its not there too
2 weeks ago
Right now,Im using good_transformed_df directly to merge with silver
good_transformed_df- which is output of transformation func to which bronze data is feb. But still its not showing up.
2 weeks ago
It is because of using temp view. To debug further you would need to write all the source tables, transformations, target tables, actual lineage and expected lineage, but as a rule of thumb if the lineage is lost when using temp view.
Lineage is captured at the table level, and it's most reliably tracked when both the source and target are referenced by table names. If you're creating a temporary view and then using that view to create a source_df which is then merged into a silver table, it's possible that the lineage tracking system is unable to capture this lineage due to the use of temporary view.
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group