topic Re: Exposing Unity Catalog lineage schema in Data Governance

Exposing Unity Catalog lineage schema

squarshie — Mon, 01 May 2023 19:01:50 GMT

Have been informed by a vendor we are using for Data Observability that the Unity Catalog lineage can be accessed via the systems schema. This is not enabled by default and we will need to execute a PUT request to the endpoint below. I am unable to find any documentation for this.

api/2.0/unity-catalog/metastores/<metastore_id>/systemschemas/lineage

Is any one familiar with this API call and have details on what exactly this call is going to do?

Re: Exposing Unity Catalog lineage schema

squarshie — Mon, 08 May 2023 06:19:27 GMT

@Debayan Mukherjee

Found out this is a preview feature and hence why you cannot find any documentation on the api website. Feature exposes a lineage table in the system schema so you can programmatically query lineage data. While you can access linage via the API, our specific use case required being able to query this information directly from a table.

Re: Exposing Unity Catalog lineage schema

Anonymous — Thu, 18 May 2023 09:41:47 GMT

Hi @Stephen Quarshie

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!

Re: Exposing Unity Catalog lineage schema

Debayan — Sun, 07 May 2023 19:32:50 GMT

Hi,

UC APIs are under API 2.1. You can get all the API details here: https://docs.databricks.com/dev-tools/api/latest/index.html and

https://docs.databricks.com/api-explorer/workspace/catalogs/list

I do not think there is an API as mentioned above.

Please tag @Debayan with your next comment so that I will get notified. Thank you!

Re: Exposing Unity Catalog lineage schema

Debayan — Thu, 11 May 2023 05:46:57 GMT

Hi, Yes, this feature is on private preview.