topic Re: Unity Catalog Lineage APIs – Are these REST APIs publicly supported? in Data Governance

Unity Catalog Lineage APIs – Are these REST APIs publicly supported?

faruk — Mon, 17 Aug 2026 07:11:04 GMT

Hi everyone,

I’m currently working on a PoC to build a data dictionary in Databricks, and one of the requirements is to retrieve lineage information programmatically.

I was advised that Databricks provides Unity Catalog Lineage APIs that can retrieve table-level and column-level lineage, following the Unity Catalog permission model and without requiring direct access to the system.access schema.

I was provided with an example endpoint:

/api/2.1/unity-catalog/lineage/table-lineages

However, after searching the current Databricks documentation, I cannot find this endpoint or any official documentation for a table-lineages REST API.

The situation is that our parent company does not allow us to have direct access to the system.access schema for security and governance reasons. Therefore, querying system.access.table_lineage is not an option for our PoC.

My goal is to build a data dictionary and enrich it with lineage information, so I’m looking for a supported REST API that would allow me to retrieve table-level lineage without accessing system.access.table_lineage directly.

Could anyone clarify:

Is /api/2.1/unity-catalog/lineage/table-lineages a supported/public Databricks REST API?
If so, where can I find the official documentation?
Is there another supported REST API to retrieve table-level lineage without querying system.access.table_lineage?
Is column-level lineage also available through a supported API?

Any guidance or documentation would be greatly appreciated.

Thanks!

Re: Unity Catalog Lineage APIs – Are these REST APIs publicly supported?

nick_martinek — Mon, 17 Aug 2026 11:40:31 GMT

One possible alternative, if the main blocker is direct access to system tables, would be for the parent company to expose a restricted view over system.access table, filtered only to the relevant catalogs/schemas. Maybe that would enable PoC work without violation of governance and security.

Re: Unity Catalog Lineage APIs – Are these REST APIs publicly supported?

balajij8 — Mon, 17 Aug 2026 12:39:47 GMT

Hi Faruk,

The referenced endpoint is generally an internal route used rather than a supported public REST API. Unity Catalog system tables is the valid option. However, if you are blocked from it and need a working alternative for the PoC right now. You can follow below

Query History Reconstruction via system.query.history - You can reconstruct write operations and table-level dependencies by extracting queries that create or modify tables and parsing their underlying SQL text using the system query history table. It tracks table creation and modification events, identifies upstream source tables via SQL parsing and helps map out write operations. It requires maintaining your own SQL parsing logic and is complex & less precise than engine-native lineage capture. You can use it for High-level impact analysis, audit trails and identifying table creators or pipeline execution metadata.

Scoped Access to System Tables - For building a production-grade data dictionary, obtaining read permissions on native lineage tables is the recommended approach. You can request your account or workspace administrator to grant SELECT privileges specifically on table_lineage and column_lineage. It provides fully automated table-level and column-level lineage without requiring broader or elevated access across other governance domains.

Re: Unity Catalog Lineage APIs – Are these REST APIs publicly supported?

ShamenParis — Mon, 17 Aug 2026 19:25:29 GMT

Hi @faruk ,

I completely understand the frustration. Navigating enterprise permissions is a headache, and I actually went down this exact same rabbit hole recently when I was trying to pull lineage data strictly using the Databricks REST API or SDK.

To give it to you straight, that /api/2.1/unity-catalog/lineage/table-lineages endpoint you were given is an internal, undocumented API used by the Databricks UI under the hood. Because it's an internal tool, there is no official public documentation for it, and using it for a production PoC is highly risky since Databricks can change or deprecate it at any time without warning. Unfortunately, there are currently no officially supported REST APIs for either table-level or column-level lineage.

The reason Databricks does this is actually by design. Over the last couple of years, they have shifted heavily toward an "everything is a table" architecture with their System Tables. Lineage graphs can get incredibly massive, and trying to paginate through hundreds of thousands of records via REST APIs just doesn't scale well. By keeping lineage inside system.access.table_lineage, they let us use the Spark distributed engine to query it and easily join it with other metadata.

When I was dealing with this, I realized giving up on the SDK and using SQL was the only sustainable way forward. I ended up running a straightforward query to just pull the direct source-to-target relationships, which looks something like this:

query = """ SELECT DISTINCT source_table_catalog AS SourceCatalog, source_table_schema AS SourceSchema, source_table_name AS SourceTable, target_table_catalog AS TargetCatalog, target_table_schema AS TargetSchema, target_table_name AS TargetTable FROM system.access.table_lineage WHERE source_table_name IS NOT NULL AND target_table_name IS NOT NULL """ lineage_df = spark.sql(query) display(lineage_df)

Since your parent company is strictly blocking your access to the system.access schema, trying to reverse-engineer an unsupported internal API is probably going to cause more problems down the line.

The most realistic workaround here is to have a conversation with your security and governance team. Instead of asking for raw access, ask them to create a "Secure View" on top of system.access.table_lineage. They can write a simple view that filters the lineage table so your PoC user or Service Principal only sees data for the specific catalogs and schemas you are actually allowed to govern, completely hiding the rest of the enterprise data. This satisfies their strict security rules while allowing you to use the officially supported architecture to build your data dictionary.

Re: Unity Catalog Lineage APIs – Are these REST APIs publicly supported?

faruk — Tue, 18 Aug 2026 05:03:22 GMT

@balajij8 Thank you for your advice, i definetly agree that for a production-grade data dictionnary i have to have the needed permissions.

Re: Unity Catalog Lineage APIs – Are these REST APIs publicly supported?

faruk — Tue, 18 Aug 2026 05:07:04 GMT

@ShamenParis Thank you for your elaborate answer. I know that it is undocumented and more over it doesn't really do what i want Databricks Lineage API retains lineage data indefinitely, even after the referenced column have been deleted and it make the work harder. Another limitation it does not provide the complete lineage graph, it give only the immediate upstream and downstream relationship.

I will try to ask them the secure View but i already know their answer.

Re: Unity Catalog Lineage APIs – Are these REST APIs publicly supported?

ShamenParis — Tue, 18 Aug 2026 06:13:32 GMT

Hi @faruk

I understand the situation, but you can show this reply to your security team and try to get a view created. Once you have that access, you can use a recursive function to get the lineage of lineage in memory.

Here is the exact logic I used to build it (You need to change the code a bit):

def _build_graph(self): print("Building in-memory dependency graph...") query = """ SELECT DISTINCT target_table_full_name AS object, target_type AS object_type, source_table_full_name AS depends_on_object, source_type AS depends_on_object_type FROM system.access.table_lineage WHERE source_type IN ('TABLE', 'VIEW') AND target_type IN ('TABLE', 'VIEW') AND target_table_full_name IS NOT NULL AND source_table_full_name IS NOT NULL """ df_lineage = self.spark.sql(query).collect() for row in df_lineage: obj = row['object'].lower() obj_type = row['object_type'] dep = row['depends_on_object'].lower() dep_type = row['depends_on_object_type'] # Store valid names self.all_known_tables.add(obj) self.all_known_tables.add(dep) # Store their types self.object_types[obj] = obj_type self.object_types[dep] = dep_type # Build relationships if obj not in self.dependency_graph: self.dependency_graph[obj] = set() self.dependency_graph[obj].add(dep) print(f"Graph built successfully. Tracking {len(self.dependency_graph)} target objects.")

The best part about doing it this way is that it solves your deleted data problem. You just cross-reference that all_known_tables set against the active system.information_schema.tables. If an object doesn't exist there anymore, you know it was deleted and can safely drop it from your dictionary.