Dataplex integration with Unity Catalog
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-20-2025 11:42 PM
Hi,
I want to import Databricks metadata information from unity catalog including lineage into GCP Dataplex governance solution. Can you let me know the how this can be achieved.
Similar to Purview and Unity Catalog integration, are there APIs to push information from Unity Catalog into Dataplex.
Thanks
Shankar
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-21-2025 04:45 AM
To import Databricks metadata information from Unity Catalog, including lineage, into GCP Dataplex, you can use the Unity Catalog APIs to extract the necessary metadata and lineage information and then push this data into Dataplex using its APIs.
Here are the steps you can follow:
-
Extract Metadata and Lineage from Unity Catalog:
- Use the Unity Catalog REST APIs to retrieve metadata and lineage information. You can find the API documentation and endpoints for Unity Catalog in the Databricks documentation. The relevant APIs include those for listing catalogs, schemas, tables, and retrieving lineage information.
-
Transform the Data:
- Transform the extracted metadata and lineage information into the format required by GCP Dataplex. This may involve converting the data into JSON or another suitable format that Dataplex can ingest.
-
Push Data to GCP Dataplex:
- Use the Dataplex APIs to push the transformed metadata and lineage information into Dataplex. The Dataplex documentation provides details on how to use their APIs for metadata management.
Currently, there is no direct integration between Unity Catalog and GCP Dataplex similar to the integration between Purview and Unity Catalog. Therefore, you will need to use the APIs provided by both platforms to achieve this integration manually.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-21-2025 05:23 AM
Thanks Walter for the information. Can you share the links for the APIs.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-21-2025 05:38 AM
Hi Walter
Is it possible to import metadata from any governance/catalog tool into unity catalog or it is only push mechanism from unity catalog into dataplex/purview?

