Hello,
I'm designing a system that combines a lake-house and a real-time path that federates across multiple databases. The requirements led us to adopt Starburst Galaxy (a cloud-hosted distributed query engine) for query federation across the various DBs and (mostly reading) into the lake-house. However, I would also like to make use of the Databricks platform to manage the lake-house and upcoming ML work. The tentative approach to fuse these two platform: let Databricks run in the typical setup (bronze-silve-gold, Delta Lake / DLT, etc.), and let Galaxy / Trino access the Unity Catalog somehow so they both see the same lake metadata and data (Delta Lake tables).
Question:
Is this (accessing Unity Catalog as Hive Metastore externally) feasible? The idea was based on this blog post: https://www.databricks.com/blog/extending-databricks-unity-catalog-open-apache-hive-metastore-api. Any additional pointers on this matter is greatly appreciated.
Thanks!
