cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Governance
Join discussions on data governance practices, compliance, and security within the Databricks Community. Exchange strategies and insights to ensure data integrity and regulatory compliance.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Dataplex integration with Unity Catalog

ShankarM
Contributor

Hi,

I want to import Databricks metadata information from unity catalog including lineage into GCP Dataplex governance solution. Can you let me know the how this can be achieved.

Similar to Purview and Unity Catalog integration, are there APIs to push information from Unity Catalog into Dataplex.

Thanks

Shankar

 

 

 

3 REPLIES 3

Walter_C
Databricks Employee
Databricks Employee

To import Databricks metadata information from Unity Catalog, including lineage, into GCP Dataplex, you can use the Unity Catalog APIs to extract the necessary metadata and lineage information and then push this data into Dataplex using its APIs.

Here are the steps you can follow:

  1. Extract Metadata and Lineage from Unity Catalog:

    • Use the Unity Catalog REST APIs to retrieve metadata and lineage information. You can find the API documentation and endpoints for Unity Catalog in the Databricks documentation. The relevant APIs include those for listing catalogs, schemas, tables, and retrieving lineage information.
  2. Transform the Data:

    • Transform the extracted metadata and lineage information into the format required by GCP Dataplex. This may involve converting the data into JSON or another suitable format that Dataplex can ingest.
  3. Push Data to GCP Dataplex:

    • Use the Dataplex APIs to push the transformed metadata and lineage information into Dataplex. The Dataplex documentation provides details on how to use their APIs for metadata management.

Currently, there is no direct integration between Unity Catalog and GCP Dataplex similar to the integration between Purview and Unity Catalog. Therefore, you will need to use the APIs provided by both platforms to achieve this integration manually.

Thanks Walter for the information. Can you share the links for the APIs.

 

Hi Walter

Is it possible to import metadata from any governance/catalog tool into unity catalog or it is only push mechanism from unity catalog into dataplex/purview?

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group