Databricks lineage

[Rachana2]

Hello,

I am trying to get the table lineage i.e upstreams and downstreams of all tables in unity catalog into my local database using API calls. I need my db to be up to date, if the lineage is updated in one of the in databricks, i have to update same in my database, but i dont want to make api calls to all tables in UC to check if its updated or not. To avoid this is there any way to get the last updated time lineage? or any other way so that i can make api calls to only those tables where were updated from last run.

[Alberto_Umana]

Hello @Rachana2,

Have you tried using system lineage tables instead?

system.access.table_lineage and system.access.column_lineage

[Rachana2]

Thankyou for your quick response!

Is there any other way we can find the last altered time of lineage apart from system.access tables?

[SantoshJoshi]

Hi @Rachana2,

As @Alberto_Umana has mentioned I'd check table_lineage / column_lineage tables, as maintaining a lineage through a bespoke pipeline/tooling may not be a right approach.

Can you please explain your use case which explains why you don't want to use the system tables? (is your Databricks not in UC?)