Databricks as a Semantic Engine: Why the Semantic Layer Was Never Enough

[ankush_a]

For twenty years, the semantic layer has been the industry's answer to a simple question: how do we make sure everyone means the same thing when they say "revenue"? And for twenty years, the answer has mostly failed. Not because the idea was wrong, but because of where we put it.

Semantic layers traditionally lived in BI tools — locked inside a Tableau workbook, a Power BI dataset, a Looker model. Each tool had its own copy of the truth, each copy drifted independently, and the business logic that mattered most was the least governed asset in the stack. When finance and marketing reported different quarterly revenue numbers, the root cause was rarely the data. It was the semantics.

The arrival of AI agents turned this from an annoyance into an existential problem. An analyst who gets an ambiguous answer knows to ask a colleague. An agent that gets ambiguous semantics confidently generates wrong SQL at machine speed. The industry consensus that emerged over the past year is blunt: AI doesn't have an intelligence problem, it has a context problem. Text-to-SQL against raw schemas plateaus well below production-grade accuracy — not because the model can't write SQL, but because the schema doesn't encode what the business actually means.

This is why I've stopped thinking about Databricks as a platform that hosts a semantic layer, and started thinking about it as a semantic engine: a system that doesn't just store business meaning, but defines it, learns it, executes it, and governs it as a single loop. That distinction — layer versus engine — is the most important architectural shift I'm seeing in enterprise data right now, and it's worth unpacking how each stage of the loop works.

A layer describes. An engine runs.

A semantic layer is a static artifact: a mapping file, a set of definitions, a model someone built eighteen months ago and hopes still reflects reality. It describes meaning but doesn't participate in how that meaning is created, validated, or consumed. It decays the moment it ships.

An engine is different. An engine takes inputs, does work, and produces outputs continuously. Applied to semantics, that means four capabilities operating together:

1. Definition — business concepts and KPIs authored once as governed, first-class objects
2. Learning — semantics that update from real usage, lineage, and behavior rather than manual curation alone
3. Execution — every consumer (SQL, BI, notebooks, agents) computing against the same definitions at query time
4. Governance — access, policy, and audit applied at the level of business meaning, not just tables

Most architectures deliver one or two of these, bolted together across vendors. What makes the Databricks approach interesting is that all four run inside the same governance boundary — Unity Catalog — and that's precisely what turns a layer into an engine.

Definition: semantics as governed objects, not documentation

The foundation is treating business meaning the way we learned to treat data: as versioned, governed, discoverable assets.

Unity Catalog Metrics let you define KPIs — revenue, churn, active users, margin — once, declaratively, and then query them consistently from SQL, dashboards, APIs, and agents. This goes well beyond simple single-table measures. Multi-fact relationships handle the star and snowflake realities of enterprise data; level-of-detail calculations compute at the granularity the business question demands rather than the granularity the table happens to have; parameterized metrics adapt to runtime inputs; window measures make period-over-period analysis a definition rather than a copy-pasted SQL pattern.

The Business Glossary adds the human vocabulary layer: authoritative definitions of business concepts, connected directly to the underlying data assets, and — critically — co-curated by humans and AI rather than maintained as a wiki page nobody reads. Domains organize all of it along business lines rather than technical ones.

The design principle here matters more than any individual feature: semantics live where the data and compute live. When the definition of "net revenue" is a governed object in the catalog rather than a formula in a dashboard, there is no drift problem to solve, because there is nothing to drift.

Learning: the ontology that maintains itself

Here's the uncomfortable truth every architect who has built a semantic layer knows: the modeling never finishes. Businesses reorganize, products launch, definitions evolve. Manually curated ontologies are obsolete on arrival, and the maintenance burden is why most semantic-layer initiatives quietly stall after the first quarter of enthusiasm.

This is where the engine framing earns its name. Genie Ontology is a continuously learned enterprise context layer: it ingests the user-defined semantic foundation from Unity Catalog, then enriches it with signals the platform already has — query patterns, lineage, usage behavior, workspace assets like notebooks and dashboards. Instead of a static model that decays, you get an ontology that adapts as the business does.

The results are measurable. Databricks' internal benchmarks put ontology-grounded natural-language querying at roughly 84.5% accuracy — a substantial jump over naive text-to-SQL approaches, which routinely land far lower against real enterprise schemas. That delta is the entire business case in one number: the same model, the same data, transformed by whether the engine supplies meaning at query time.

There's a virtuous cycle here worth naming explicitly. Human-curated semantics make the learned ontology more accurate; the learned ontology surfaces gaps and suggests definitions that humans then certify. Neither pure manual modeling nor pure inference gets you there. The loop does.

Execution: one definition, every consumer — including agents

Definitions and learning only matter if consumption is unified. This is where most multi-vendor semantic stacks fracture: the BI tool computes the metric one way, the notebook another, the agent a third.

In the engine model, execution is centralized. A metric defined in Unity Catalog returns the same answer whether it's queried from Databricks SQL, rendered in a dashboard, called through an API, or invoked by an agent. For the agentic path specifically, managed MCP servers expose governed data to any MCP-compatible agent: structured data through Genie, unstructured content through Vector Search, custom business logic through Unity Catalog functions. The agent doesn't get raw tables and a prayer — it gets certified concepts, defined relationships, and governed measures.

For solution architects, this reframes a familiar conversation. The question is no longer "which semantic layer product should sit between our lakehouse and our BI tools?" It's "which definitions do we certify in the engine, and which consumers do we point at them?" Third-party semantic layers still have a genuine role — cross-platform ontologies spanning multiple clouds and warehouses, or MDX/DAX interfaces for deeply entrenched Excel and Power BI estates — but they increasingly consume from the engine rather than replace it.

A working example: "net revenue" from definition to agent

Abstractions are easier to trust when you can see the mechanics, so let's build one metric through the whole engine. Take the classic offender: net revenue. Ask three teams and you'll get three formulas — gross minus discounts, minus returns, minus both, maybe minus freight. Here's how the engine ends that argument.

Step 1 — Define it once, as a governed object. A Metric View in Unity Catalog is declared in YAML: source, joins, dimensions, and measures, with descriptions and synonyms that both humans and agents reason over.

version: 1.0

source: sales.gold.fact_orders

joins:
  - name: customer
    source: sales.gold.dim_customer
    on: source.customer_id = customer.customer_id

dimensions:
  - name: Order Month
    expr: DATE_TRUNC('MONTH', source.order_date)
  - name: Region
    expr: customer.region
  - name: Sales Channel
    expr: source.channel

measures:
  - name: Net Revenue
    expr: SUM(source.gross_amount - source.discount_amount - source.return_amount)
    comment: >
      Gross order value less discounts and returns. Excludes freight
      and taxes. This is the certified definition used in executive
      reporting. Synonyms: net sales, revenue net of returns.
  - name: Net Revenue Prior Month
    expr: SUM(source.gross_amount - source.discount_amount - source.return_amount)
    window:
      - order: Order Month
        range: trailing 1 month
        semiadditive: last

Create it with CREATE VIEW sales.gold.mv_revenue_metrics WITH METRICS LANGUAGE YAML AS ..., and it's now a first-class Unity Catalog asset — permissioned, lineage-tracked, versioned, discoverable.

Step 2 — Every consumer computes the same thing. An analyst in Databricks SQL queries it with the MEASURE() function:

SELECT
  `Region`,
  `Order Month`,
  MEASURE(`Net Revenue`)  AS net_revenue
FROM sales.gold.mv_revenue_metrics
GROUP BY `Region`, `Order Month`
ORDER BY `Order Month` DESC;

Notice what the analyst didn't do: repeat the formula. The definition is resolved by the engine at query time. A dashboard pointed at the same metric view renders the same number. If finance later decides freight should be excluded differently, one YAML change propagates everywhere — there is no second copy to forget.

Step 3 — The agent gets meaning, not guesswork. Now a business user asks Genie: "How did net revenue trend by region last quarter?" Instead of staring at gross_amount, discount_amount, and return_amount and guessing at a formula, Genie resolves "net revenue" to the certified measure — helped by the comment and synonyms — and generates SQL against the metric view. An external agent connecting through the managed MCP server takes the same path: it discovers the governed measure, invokes it, and inherits the caller's Unity Catalog permissions, so a regional manager's agent only ever aggregates rows their role permits.

Step 4 — The learning loop closes. Suppose users keep asking about "revenue per active distributor" — a concept nobody modeled. The ontology infers it from query patterns and notebook logic, and surfaces it as a learned concept with its lineage. A domain owner reviews the inferred definition, corrects the denominator, and promotes it into the Glossary and a metric view. What began as unmodeled folk knowledge becomes a certified, governed object — and every agent immediately gets smarter for it.

That's the engine in miniature: one declaration, uniform execution across SQL, BI, and agents, governance inherited rather than reimplemented, and a feedback loop that grows the semantic surface where the business actually asks questions. (Syntax evolves between releases, so treat the YAML above as illustrative and check the current Metric Views documentation before deploying.)

Governance: policy at the level of meaning

The final stage is the one that makes the whole thing enterprise-viable. Because every semantic object lives in Unity Catalog, the governance model you already run — access controls, lineage, audit — applies to meaning itself. Who can see the margin metric. Which agent is allowed to invoke which function. Where a KPI's inputs came from, column by column.

This is the piece that pure-play semantic tools structurally cannot offer: governance of semantics and governance of data as one system. When an executive asks "can I trust this number?", lineage from the dashboard through the metric definition to the source tables answers it. When a regulator asks "what did the agent access?", the audit trail is native, not reconstructed.

What this means for your architecture

If you're building on Databricks — or evaluating it — the semantic engine framing suggests three practical moves:

Certify your top twenty metrics first. Don't boil the ontology ocean. Take the KPIs that appear in executive reporting, define them as Unity Catalog Metrics, and deprecate the BI-tool copies. This is weeks of work with immediate consistency payoff, and it seeds the learned ontology with high-signal ground truth.

Treat Genie accuracy as a semantics metric, not a model metric. If natural-language answers are wrong, resist the instinct to blame the LLM. Audit the semantic foundation: missing glossary terms, ambiguous joins, uncertified metrics. In my experience, accuracy problems are context problems wearing a model costume.

Design agents against the engine, not the schema. Any agent architecture that hands raw table DDL to a model is accumulating semantic debt. Route agent access through governed interfaces — Genie for structured queries, Vector Search for unstructured retrieval, UC functions for business logic — so that every improvement to your semantics improves every agent simultaneously.

The layer is dead; long live the engine

The semantic layer failed as a product category because it was an artifact in a world that needed a process. What's emerging on Databricks is that process made concrete: definitions as governed objects, an ontology that learns, execution unified across humans and agents, and governance applied to meaning itself.

The organizations that internalize this shift won't just have more consistent dashboards. They'll have the substrate that makes enterprise AI actually work — because an agent is only as trustworthy as the semantics beneath it, and semantics are only trustworthy when they're run by an engine, not filed in a layer.