topic Re: Exploiting Hive Metastore of Databricks for lineage in Data Governance

Exploiting Hive Metastore of Databricks for lineage

NOOR_BASHASHAIK — Thu, 17 Mar 2022 15:00:21 GMT

Hi all

Wanted to check if anyone has made an attempt to exploit the Hive Metastore of Databricks for lineage?

For example, I loaded metadata of 2 databricks databases using the Collibra Marketplace provided Databricks driver. Here is the scenario -

Database 1 > Table_A

Database 2 > View_A based on Table_A

As the table & view relations are implicit, I expected the driver to show lineage/links between these 2 objects across databases within Collibra but it did not.

So, I plan to fetch the relationships information from Hive Metastore and feed into Collibra.

Couple of questions -

Where can I see the data model of Hive Metastore? Is there any documentation link from Databricks side so i can quickly understand the schemata of the metastore.
Is it advisable to query the metastore tables or are there any side-effects?
How easy will it be to fetch the relationships between tables & views? Is there an out-of-the-box query?

Re: Exploiting Hive Metastore of Databricks for lineage

Anonymous — Tue, 17 May 2022 05:43:36 GMT

Hi @NOOR BASHASHAIK (Customer) , Just a friendly follow-up. Do you still need help, or did Kaniz's response help you find the solution?

Re: Exploiting Hive Metastore of Databricks for lineage

NOOR_BASHASHAIK — Tue, 17 May 2022 06:52:05 GMT

@Chetan Kardekar @Kaniz Fatma yes, I still need a standard way (through SQL) to access Hive Metastore.

Re: Exploiting Hive Metastore of Databricks for lineage

jose_gonzalez — Tue, 07 Jun 2022 16:48:43 GMT

Hi @NOOR BASHA SHAIK,

Just a friendly follow-up. Are you still looking for help or did Kaniz's reply helped you to resolved your question?