topic Re: Databricks UC enabled but Lineage not found for one table in Data Engineering

Databricks UC enabled but Lineage not found for one table

varshini_reddy — Fri, 06 Sep 2024 04:59:37 GMT

Databricks UC enabled but Lineage not found for one table whereas i can see the lineage for the other two, any idea on why is it?. Im performing few transformations to bronze data , taking good_data_transformed as a dataframe, creating a temp view for that, using that temp view, i create a source_df and merge into silver table.

Re: Databricks UC enabled but Lineage not found for one table

AndySkinner — Fri, 06 Sep 2024 09:18:50 GMT

Is this from the UI or from the lineage table?

Probably worth checking both. The database option is in system catalog, so system.access.table_lineage

Re: Databricks UC enabled but Lineage not found for one table

varshini_reddy — Fri, 06 Sep 2024 11:12:16 GMT

i cant see in lineage in UI

Re: Databricks UC enabled but Lineage not found for one table

AndySkinner — Fri, 06 Sep 2024 11:30:46 GMT

Like I said, worth checking the database too.....

Re: Databricks UC enabled but Lineage not found for one table

varshini_reddy — Fri, 06 Sep 2024 11:46:20 GMT

its not there too

Re: Databricks UC enabled but Lineage not found for one table

varshini_reddy — Fri, 06 Sep 2024 11:48:25 GMT

Right now,Im using good_transformed_df directly to merge with silver

good_transformed_df- which is output of transformation func to which bronze data is feb. But still its not showing up.

Re: Databricks UC enabled but Lineage not found for one table

filipniziol — Fri, 06 Sep 2024 14:47:47 GMT

It is because of using temp view. To debug further you would need to write all the source tables, transformations, target tables, actual lineage and expected lineage, but as a rule of thumb if the lineage is lost when using temp view.

Lineage is captured at the table level, and it's most reliably tracked when both the source and target are referenced by table names. If you're creating a temporary view and then using that view to create a source_df which is then merged into a silver table, it's possible that the lineage tracking system is unable to capture this lineage due to the use of temporary view.