topic Databricks lineage in Get Started Discussions

Databricks lineage

Rachana2 — Thu, 06 Mar 2025 10:54:29 GMT

Hello,

I am trying to get the table lineage i.e upstreams and downstreams of all tables in unity catalog into my local database using API calls. I need my db to be up to date, if the lineage is updated in one of the in databricks, i have to update same in my database, but i dont want to make api calls to all tables in UC to check if its updated or not. To avoid this is there any way to get the last updated time lineage? or any other way so that i can make api calls to only those tables where were updated from last run.

Re: Databricks lineage

Alberto_Umana — Thu, 06 Mar 2025 13:53:45 GMT

Hello @Rachana2,

Have you tried using system lineage tables instead?

system.access.table_lineage and system.access.column_lineage

Re: Databricks lineage

Rachana2 — Fri, 07 Mar 2025 18:56:29 GMT

Thankyou for your quick response!

Is there any other way we can find the last altered time of lineage apart from system.access tables?

Re: Databricks lineage

SantoshJoshi — Sat, 08 Mar 2025 23:17:22 GMT

Hi @Rachana2,

As @Alberto_Umana has mentioned I'd check table_lineage / column_lineage tables, as maintaining a lineage through a bespoke pipeline/tooling may not be a right approach.

Can you please explain your use case which explains why you don't want to use the system tables? (is your Databricks not in UC?)