Accessing Unity Catalog Externally (Outside of Databricks) as Hive Metastore

[illumety-jh]

Hello,

I'm designing a system that combines a lake-house and a real-time path that federates across multiple databases. The requirements led us to adopt Starburst Galaxy (a cloud-hosted distributed query engine) for query federation across the various DBs and (mostly reading) into the lake-house. However, I would also like to make use of the Databricks platform to manage the lake-house and upcoming ML work. The tentative approach to fuse these two platform: let Databricks run in the typical setup (bronze-silve-gold, Delta Lake / DLT, etc.), and let Galaxy / Trino access the Unity Catalog somehow so they both see the same lake metadata and data (Delta Lake tables).

Question: 
Is this (accessing Unity Catalog as Hive Metastore externally) feasible? The idea was based on this blog post: https://www.databricks.com/blog/extending-databricks-unity-catalog-open-apache-hive-metastore-api. Any additional pointers on this matter is greatly appreciated.

Thanks!

[Kaniz_Fatma]

Hi @illumety-jh, Yes, it is feasible to access the Unity Catalog as a Hive Metastore externally. The Unity Catalog in Databricks is designed to be additive, meaning it can be used alongside the per-workspace Hive Metastore. This setup allows you to query the Hive Metastore within the Unity Catalog, making it accessible for....

To achieve this, you can refer to the Hive Metastore as a top-level catalog called hive_metastore in the three-level namespace. 

[illumety-jh]

Thank you @Kaniz_Fatma. 

[Kaniz_Fatma]

Hi @illumety-jh, Thank you for reaching out to our community! We're here to help you.

To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedback not only helps us assist you better but also benefits other community members who may have similar questions in the future.

If you found the answer helpful, consider giving it a kudo. If the response fully addresses your question, please mark it as the accepted solution. This will help us close the thread and ensure your question is resolved.

We appreciate your participation and are here to assist you further if you need it!