Accessing Unity Catalog Externally (Outside of Databricks) as Hive Metastore

illumety-jh — Thu, 25 Jul 2024 01:27:05 GMT

Hello,

I'm designing a system that combines a lake-house and a real-time path that federates across multiple databases. The requirements led us to adopt Starburst Galaxy (a cloud-hosted distributed query engine) for query federation across the various DBs and (mostly reading) into the lake-house. However, I would also like to make use of the Databricks platform to manage the lake-house and upcoming ML work. The tentative approach to fuse these two platform: let Databricks run in the typical setup (bronze-silve-gold, Delta Lake / DLT, etc.), and let Galaxy / Trino access the Unity Catalog somehow so they both see the same lake metadata and data (Delta Lake tables).

Question:
Is this (accessing Unity Catalog as Hive Metastore externally) feasible? The idea was based on this blog post: https://www.databricks.com/blog/extending-databricks-unity-catalog-open-apache-hive-metastore-api. Any additional pointers on this matter is greatly appreciated.

Thanks!

Re: Accessing Unity Catalog Externally (Outside of Databricks) as Hive Metastore

illumety-jh — Tue, 30 Jul 2024 14:42:30 GMT

Thank you @Retired_mod.

Re: Accessing Unity Catalog Externally (Outside of Databricks) as Hive Metastore

Sijay — Wed, 13 Nov 2024 00:41:48 GMT

Hello there!

So any updates so far? Also interesting in this solution.

topic Re: Accessing Unity Catalog Externally (Outside of Databricks) as Hive Metastore in Warehousing & Analytics

Accessing Unity Catalog Externally (Outside of Databricks) as Hive Metastore

Re: Accessing Unity Catalog Externally (Outside of Databricks) as Hive Metastore

Re: Accessing Unity Catalog Externally (Outside of Databricks) as Hive Metastore