Exploiting Hive Metastore of Databricks for lineage

[NOOR_BASHASHAIK]

Hi all

Wanted to check if anyone has made an attempt to exploit the Hive Metastore of Databricks for lineage?

For example, I loaded metadata of 2 databricks databases using the Collibra Marketplace provided Databricks driver. Here is the scenario -

Database 1 > Table_A

Database 2 > View_A based on Table_A

As the table & view relations are implicit, I expected the driver to show lineage/links between these 2 objects across databases within Collibra but it did not.

So, I plan to fetch the relationships information from Hive Metastore and feed into Collibra.

Couple of questions -

1. Where can I see the data model of Hive Metastore? Is there any documentation link from Databricks side so i can quickly understand the schemata of the metastore.
2. Is it advisable to query the metastore tables or are there any side-effects?
3. How easy will it be to fetch the relationships between tables & views? Is there an out-of-the-box query?

[Anonymous]

Hi @NOOR BASHASHAIK (Customer)​ , Just a friendly follow-up. Do you still need help, or did Kaniz's response help you find the solution?

[NOOR_BASHASHAIK]

@Chetan Kardekar​ @Kaniz Fatma​ yes, I still need a standard way (through SQL) to access Hive Metastore.

[jose_gonzalez]

Hi @NOOR BASHA SHAIK​,

Just a friendly follow-up. Are you still looking for help or did Kaniz's reply helped you to resolved your question?