Hi @faruk ,

I completely understand the frustration. Navigating enterprise permissions is a headache, and I actually went down this exact same rabbit hole recently when I was trying to pull lineage data strictly using the Databricks REST API or SDK.

To give it to you straight, that /api/2.1/unity-catalog/lineage/table-lineages endpoint you were given is an internal, undocumented API used by the Databricks UI under the hood. Because it's an internal tool, there is no official public documentation for it, and using it for a production PoC is highly risky since Databricks can change or deprecate it at any time without warning. Unfortunately, there are currently no officially supported REST APIs for either table-level or column-level lineage.

The reason Databricks does this is actually by design. Over the last couple of years, they have shifted heavily toward an "everything is a table" architecture with their System Tables. Lineage graphs can get incredibly massive, and trying to paginate through hundreds of thousands of records via REST APIs just doesn't scale well. By keeping lineage inside system.access.table_lineage, they let us use the Spark distributed engine to query it and easily join it with other metadata.

When I was dealing with this, I realized giving up on the SDK and using SQL was the only sustainable way forward. I ended up running a straightforward query to just pull the direct source-to-target relationships, which looks something like this:

query = """
SELECT DISTINCT
    source_table_catalog AS SourceCatalog, 
    source_table_schema AS SourceSchema, 
    source_table_name AS SourceTable,
    target_table_catalog AS TargetCatalog, 
    target_table_schema AS TargetSchema, 
    target_table_name AS TargetTable
FROM system.access.table_lineage
WHERE source_table_name IS NOT NULL 
  AND target_table_name IS NOT NULL
"""

lineage_df = spark.sql(query)
display(lineage_df)