Unity Catalog Metric Views to be accessible to Custom Apps outside of DBX environment

[vkondepati]

Hello Everyone,

We have a requirement to expose Databricks Unity Catalog Metric Views to custom applications running outside the Databricks environment.

The application team would like to consume governed KPIs and metrics through APIs such as REST or GraphQL, rather than accessing them directly through Databricks applications or SQL interfaces.

I’d like to understand the recommended Databricks architecture for this use case:

1. Does Databricks provide an out-of-the-box REST or GraphQL API for querying Unity Catalog Metric Views from external applications?
2. If not, is the recommended approach to build a custom API/service layer that queries the Metric Views through Databricks SQL?
3. For applications requiring high concurrency and millisecond-level response times, would it make sense to materialize or synchronize the calculated metric data into Lakebase and expose it through an application/API layer?
4. Would this introduce duplication between the governed Metric Views in Unity Catalog and the serving layer in Lakebase? If so, what is the recommended approach for keeping the metric definitions and serving data consistent?
5. From an architecture perspective, which pattern would Databricks recommend:
   Custom App → API → Databricks SQL → UC Metric Views
   or
   Custom App → API → Lakebase ← Metric Views / Databricks pipelines?
6. Are there other Databricks-native capabilities better suited for providing low-latency, highly concurrent access to governed metrics for external applications?

The main objective is to maintain Unity Catalog Metric Views as the governed semantic/metrics layer, while providing external applications with a scalable, low-latency API-based consumption model.

Regards,
Venkat

[szymon_dybczak · Accepted solution]

Hi @vkondepati ,

Hi  ,

Hi,

1. Out-of-the-box REST/GraphQL for metric views?

Databricks does not appear to provide a Metric View-specific REST or GraphQL query API. Metric Views are queried through the Databricks SQL engine using SQL, including the MEASURE() function for measures.

However, external applications do not necessarily need a custom database connector. Databricks provides the SQL Statement Execution REST API, which can execute SQL against a SQL warehouse and return the result. Therefore, an external API/service can query Metric Views through this API.

So the basic pattern would be:

Custom Application → Application/API layer → Databricks Statement Execution API → SQL Warehouse → UC Metric View

2. Custom API/service layer?

For workloads where interactive analytical latency is acceptable, I would keep the Metric View as the authoritative semantic layer and query it directly through Databricks SQL. Metric Views now also support native materialization. Check docs.

Materialization for metric views | Databricks on AWS

3. Latency/Concurrency

Lakehouse//RT is currently Beta and is specifically positioned for low-latency, high-concurrency analytical workloads, including serving analytical data to custom applications. Databricks documents sub-second SQL reads for hundreds to thousands of concurrent users, and Metric Views are explicitly supported. 
If you need such high QPS then it's worth a try.

4. Lakebase

Lakebase is better aligned with operational application-serving patterns: millisecond-style key lookups, transactional workloads, large numbers of application connections, or situations where application data and analytical data need to be queried together.

Databricks provides synced tables, which replicate Unity Catalog tables, views, and materialized views into Lakebase Postgres specifically for low-latency application access. The synchronization is managed through Lakeflow pipelines.

In addition, Lakebase now provides a Data API, a PostgREST-compatible REST interface that automatically generates REST endpoints from the Postgres schema. This can potentially remove the need to build a simple CRUD/read API service at all.

[vkondepati]

Hi @szymon_dybczak,

Thanks a lot for your response and comments.
 

So the basic pattern would be:

Custom Application → Application/API layer → Databricks Statement Execution API → SQL Warehouse → UC Metric View

We still need to develop the API layer and use Databricks Statement Execution API to get access to the data.

Just want to confirm.

Regards,
Venkat

[szymon_dybczak · Accepted solution]

Hi @vkondepati ,

Yes, you get it right. If you need to improve performance then you can try to use materialized metrics views  🙂 

[vkondepati]

Is this in the future roadmap for Databricks team to consider?
This is annoying to build. It would be good to have it from DBX itself to have an option to expose them.

I was reviewing AtScale and Cube or even Denodo who support as part of their platform. I know I am comparing with other products here, but it would be good if DBX team considers it.