Databricks Metric Views: Why Gold Tables Are No Longer the End of the Pipeline

[AmitDECopilot]

In many data platforms, the problem is not that we do not have enough tables.

Usually, we have too many.

Bronze tables.
Silver tables.
Gold tables.
Reporting views.
Power BI calculations.
Tableau logic.
Excel extracts.
Ad-hoc SQL.
Notebook logic.

After some time, the same metric starts living in five different places.

Revenue is calculated one way in one dashboard.
Active customer is calculated another way in another report.
Submission count, claim count, premium, conversion rate, loss ratio  slowly every metric gets its own version depending on who built the report.

And then the real issue starts.

The pipeline may be technically correct, but business users still ask:

Why does dashboard A not match dashboard B?

This is where I think Databricks Metric Views can become important.

Not because it is just another Databricks feature, but because it gives teams a governed way to define business metrics closer to the data layer.

Databricks describes Metric Views as the core implementation of Unity Catalog semantics. They provide a centralized way to define and manage business metrics by separating measures from the dimensions used to group, filter, and aggregate them.

The real issue is not always pipeline failure

In my experience, many reporting issues are not because the pipeline failed.

The data loaded.
The job ran.
The Gold table was refreshed.
The dashboard opened.

But still the number does not match.

Why?

Because the metric logic is sitting downstream in different tools.

One team defines revenue as:

SUM(gross_amount)

Another team defines it as:

SUM(gross_amount - discount_amount)

Another team excludes cancelled records.

SUM(gross_amount - discount_amount)

WHERE cancelled_flag = 'N'

All three queries may run successfully.

But only one may be the approved business definition.

That is the gap.

A technically correct pipeline can still create confusion if downstream metric definitions are not governed.

Where Metric Views fit

My simple way to think about it:

Tables store data.
Pipelines prepare data.
Metric Views define how the business should measure data.

Gold tables are still important. I am not saying Metric Views replace Gold tables.

But Gold tables alone may not be enough anymore.

A Gold table gives curated data.
A Metric View gives governed business meaning on top of that curated data.

For example, instead of every dashboard calculating revenue differently, we can define a certified revenue metric once and reuse it across SQL, dashboards, notebooks, alerts, and AI/BI use cases.

Databricks also says Metric Views are Unity Catalog securable objects, which means they follow the same permission model as other Unity Catalog views.

That is important because metric logic should not be hidden only inside a dashboard file or one analyst’s notebook.

Example

Suppose we have a Gold table:

gold_sales_order

Columns:

order_id

customer_id

order_date

region

product_category

gross_amount

discount_amount

cancelled_flag

Now instead of repeating revenue logic everywhere, we define the business-approved metric once.

Conceptually, it may look like this:

version: 1.1

source: gold_sales_order

dimensions:

  - name: order_date

    expr: order_date

  - name: region

    expr: region

  - name: product_category

    expr: product_category

measures:

  - name: net_revenue

    expr: SUM(gross_amount - discount_amount)

    comment: Net revenue after discount, excluding cancelled orders.

Now consumers do not need to guess the revenue formula.

They use the certified metric.

That is a very simple but powerful shift.

How this helps existing pipelines

For existing pipelines, I would not start by converting everything.

That is where many platform initiatives fail.

Instead, I would start with the metrics that create the most reconciliation noise.

For example:

Revenue

Active customer

Submission count

Claim count

Premium

Renewal rate

Loss ratio

Conversion rate

Pick the top 10 metrics that business users keep questioning.

Then ask:

Where is this metric calculated today?
Is it in Power BI?
Is it in Tableau?
Is it in SQL?
Is it in Excel?
Is it in multiple reports with different logic?

Once we know that, move the approved definition into a Metric View.

This helps existing pipelines in a few ways:

1. Less duplicate logic
   Teams stop rewriting the same formula in every report.
2. Better reconciliation
   If two reports mismatch, first check whether both are using the same metric definition.
3. Cleaner governance
   Metric logic becomes visible and managed, instead of hidden inside dashboards.
4. Better migration path
   Existing dashboard logic can slowly move into a governed Databricks layer.
5. Better AI/BI readiness
   AI tools need business context. Metric Views give AI/BI and Genie a more governed path instead of asking AI to guess from table and column names. Databricks says Metric Views can be reached from Databricks Notebooks, SQL editor, AI/BI Dashboards, and AI/BI Genie.

How this helps new pipelines

For new pipelines, I would make Metric Views part of the design from the beginning.

A pipeline should not stop at:

Gold table created.

It should move toward:

Gold table , data quality checks ,metric definitions , ownership , documentation ,semantic context.

A practical new pipeline flow can be:

1. Capture business metrics during requirement phase

2. Build Bronze / Silver / Gold tables

3. Add data quality and reconciliation checks

4. Define Metric Views on top of Gold tables

5. Add business descriptions and ownership

6. Validate with business users

7. Expose to BI, SQL, dashboards, and AI/BI

This is a better end state.

Because business users do not only care that a table exists.

They care whether the number is trusted.

Why this matters more with AI/BI

Earlier, a human analyst would know hidden business rules.

They may know:

Use this table for executive reporting.
Exclude these records.
Use fiscal month, not calendar month.
Use accounting date, not transaction date.
Do not use this old dashboard logic.

But when an AI assistant is answering questions, it needs that context in a structured way.

If we expose 500 tables and weak metadata to AI, it may pick the wrong table, wrong column, wrong grain, or wrong filter.

That is dangerous because the answer may still look confident.

So my view is:

AI analytics does not reduce the need for metric governance.
It increases the need for metric governance.

Metric Views can become one of the building blocks for AI-ready analytics because they provide governed metrics and business semantics closer to the platform.

But Metric Views alone will not solve everything

This is important.

A Metric View is not magic.

If the business definition is wrong, the Metric View will only standardize the wrong logic.

So each important metric still needs:

Business definition

Metric owner

Source table

Grain

Filters / exclusions

Refresh expectation

Data quality checks

Example query

Business sign-off

Certification status

Without this, Metric Views can become just another layer.

With ownership and validation, they can become a real semantic consumption layer.

My practical adoption approach

I would adopt Metric Views in phases.

Phase 1: Identify noisy metrics

Find the top 10 metrics causing mismatch across dashboards.

Phase 2: Map current logic

Check where each metric is currently defined dashboards, SQL, notebooks, Excel, reports.

Phase 3: Agree on business definition

Get business and data owners to confirm the correct calculation.

Phase 4: Create Metric Views

Move the approved logic into Databricks Metric Views.

Phase 5: Migrate consumers slowly

Do not break everything at once. Start with one dashboard or one domain.

Phase 6: Measure adoption

Track:

Number of dashboards using certified metrics

Reduction in duplicate formulas

Reduction in reconciliation issues

Faster onboarding for new reports

Improved AI/BI answer consistency

This is how adoption becomes practical.

Final thought

For me, Databricks Metric Views are not just a BI feature.

They represent a change in how we think about pipeline completion.

Earlier, many teams thought:

Pipeline is done when the Gold table is ready.

Now I think the better standard is:

Pipeline is done when the Gold table, data quality rules, certified metrics, and business context are ready.

That is especially important as AI/BI adoption increases.

Because AI does not need access to more random tables.

It needs trusted business context.

My takeaway:

Gold tables prepare the data.
Metric Views help define how the business should measure the data.

And that metric layer may become one of the most important parts of modern Databricks pipeline design.

[savlahanish27 · Accepted solution]

The problem you're describing with metric definitions living in five different places is exactly what pushed us to add a Views layer on top of Gold in a recent SAP to Databricks migration.

We had three Tableau workbooks calculating vendor aging. One using 30/60/90 days buckets. One using calendar months. One with a bug where credits were being included in the overdue bucket. All three were being treated as correct by different teams. The fix was moving the approved aging logic into a single view in the platform - one definition, consumed by all three tools.

That solved the immediate problem, but it was still a manual process. We had to decide which logic was "approved," document it in the view comment, and trust that new consumers would use the view rather than recalculate from Gold directly. Nothing in the platform enforced it.

Metric Views feels like the platform-level answer to exactly that gap - instead of a view that happens to have the right logic inside it, you have a certified metric object that Unity Catalog can govern, permission, and expose to AI/BI tools with the business context attached. The point you make about AI needing governed metrics rather than just table access is the one that stands out most to me. An AI tool guessing revenue logic from column names is a different category of risk than a human analyst doing the same thing, because the AI answer looks confident regardless.

Curious how you're thinking about the transition period where some dashboards are using certified Metric Views and others are still pulling from Gold directly - in our case that overlap window was where the most confusion happened, because suddenly you have three versions instead of two.