cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Databricks UC enabled but Lineage not found for one table

varshini_reddy
New Contributor III

Databricks UC enabled but Lineage not found for one table whereas i can see the lineage for the other two, any idea on why is it?. Im performing few transformations to bronze data , taking good_data_transformed as a dataframe, creating a temp view for that, using that temp view, i create a source_df and merge into silver table. 

6 REPLIES 6

AndySkinner
New Contributor III

Is this from the UI or from the lineage table?

Probably worth checking both.  The database option is in system catalog, so system.access.table_lineage

i cant see in lineage in UI

Like I said, worth checking the database too.....

its not there too

varshini_reddy
New Contributor III

Right now,Im using  good_transformed_df directly to merge with silver

good_transformed_df- which is output of transformation func to which bronze data is feb. But still its not showing up. 

filipniziol
Contributor

It is because of using temp view. To debug further you would need to write all the source tables, transformations, target tables, actual lineage and expected lineage, but as a rule of thumb if the lineage is lost when using temp view.

Lineage is captured at the table level, and it's most reliably tracked when both the source and target are referenced by table names. If you're creating a temporary view and then using that view to create a source_df which is then merged into a silver table, it's possible that the lineage tracking system is unable to capture this lineage due to the use of temporary view.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group