New board topics in Resources

Announcing Databricks AppQuest: Hands-On Guidance and Cash Prizes for APJ Developers

epandya — Fri, 11 Sep 2026 17:43:07 GMT

Databricks and AngelHack present Databricks AppQuest, a five-quest program that helps APAC developers build and ship working AI apps faster, with hands-on guidance from the Databricks team and a US$10,000 prize pool.

CUSTOMER STORY | Siemens Healthineers modernizes MRI scanner data on Databricks

Tushar_Parekar — Fri, 11 Sep 2026 17:04:19 GMT

“A straight move to the cloud would have cost two to four times more, so we would never have done it. The design with Databricks came out cheaper, and now we get the AI capabilities on top.” - Michael Kelm, Cloud Data Platform Lead, Siemens Healthineers

Siemens Healthineers collects roughly 100 TB of Magnetic Resonance Imaging (MRI) scanner data every month to support service, stability and compliance. After two decades, its on-premises platform was becoming harder and more expensive to scale, while sharing historical data across teams could take weeks or months.

The company rebuilt its XMART platform on Databricks with Delta Lake and Unity Catalog, moving beyond lift-and-shift to a governed data-product architecture. The result: lower costs, permission-based sharing instead of repeated data copies, and a foundation for self-service analytics with Genie One.

Key highlights:

Around 50% lower data platform costs than the previous on-premises setup.
100 TB of MRI data processed monthly on Databricks.
Weeks to near-instant historical access through governed permission grants rather than exports and transfers.
80% and 90% cost reductions achieved through targeted optimization of two data products.
A path to self-service analytics for 1,000+ users, with Siemens Healthineers preparing to extend access through Genie One.

🔗 Check out the full story 👈

Announcement | Autoscaling Lakebase Postgres

Tushar_Parekar — Fri, 11 Sep 2026 15:19:29 GMT

Choosing a fixed database size before you know the workload is an old pattern. Lakebase Postgres removes that sizing exercise by continuously adjusting compute to match demand, while keeping PostgreSQL running and the data safely decoupled in the storage layer.

Key highlights

Separate compute from durable storage: Lakebase runs PostgreSQL on stateless compute while safekeepers, pageservers, and object storage preserve WAL, page versions, and history. Because compute owns no durable data, it can be resized, moved, restarted, or replaced without moving the database underneath it.
Use three signals, not CPU alone: The autoscaling algorithm tracks CPU load, memory use, and the frequently accessed working set. The final target is the largest of those goals, helping Lakebase respond not only to processor pressure but also to memory constraints and cache misses. See the autoscaling documentation.
Estimate the working set over time: A time-aware cardinality estimate helps distinguish the workload that is active now from an old burst that has already ended. This lets the system protect useful cache without keeping compute oversized indefinitely.
Resize a live database: Autoscaler agents, VM monitors, the Kubernetes scheduler, and NeonVM coordinate to add or remove CPU and memory from a running VM. If a node cannot accommodate an upscale, the VM can move to another node while keeping its IP address and existing connections open.
Scale down as deliberately as you scale up: Downscaling is checked against PostgreSQL and guest memory requirements before capacity is removed. Within the configured range, autoscaling adjusts compute without restarts or connection interruptions, reducing cost during quieter periods.

👉 Read the full story here

Building for Failure: Implementing Data Quality Firewalls in Petabyte-Scale Medallion Architectures

Khasim_1 — Fri, 11 Sep 2026 11:42:24 GMT

In my 13 years of architecting data platforms—from Retail to Industrial Gas Power—I’ve learned one universal truth: A fast pipeline that delivers bad data is just a liability.

As we move toward Declarative Pipelines (DLT), the role of the Architect shifts from "building the move" to "protecting the data." Here is how I approach building Data Quality Firewalls at scale.

The "Expectation" Framework: Using DLT expectations is a game-changer, but it requires a strategy. I categorize expectations into three tiers:

Critical (Fail): Schema violations or missing primary keys that would corrupt downstream logic.
Warning (Drop): Records that are directionally useful but technically flawed (e.g., negative inventory counts).
Monitored (Alert): Data that is technically valid but falls outside of historical norms.

The "Quarantine" Pattern: A major pitfall in large-scale migrations is "record loss." We don't just drop data; we quarantine it. By capturing failed expectations into a governed "Invalid" table with full lineage, we enable data stewards to correct the source without breaking the pipeline flow.
The ROI of Reliability: Investing in these firewalls at the Bronze-to-Silver transition reduces the cost of "Day 2" debugging by up to 70%. When your data is governed by Unity Catalog and protected by declarative quality gates, you don't just have a pipeline; you have a trusted asset.

Where Omnigent Fits Alongside ChatGPT Work, Claude Cowork and Cursor Projects

MouR — Fri, 11 Sep 2026 09:51:41 GMT

If you use ChatGPT Work, Claude Cowork or Cursor Projects, much of Omnigent will look familiar.

All of these products now organize work around projects, files and persistent context. They can run tasks for longer periods, connect to external tools and let agents do more than answer a single prompt.

ChatGPT Projects keep conversations, instructions and files together. ChatGPT Work can take a larger assignment, work across connected applications and produce finished deliverables.

Claude Projects provide a similar place for files, instructions and memory. Cowork can run tasks in the cloud, use connectors and plugins, work with the browser or computer and continue across devices.

Cursor Projects is centered on software development. It keeps repository context, coordinates cloud agents and connects the work with GitHub, testing, CI and pull requests.

So when Omnigent opens with another project-like interface, sessions, files, agents and cloud execution, the obvious reaction is:

Don't we already have this?

In many cases, yes.

If a team works mainly inside Claude, Cursor or OpenAI, those environments may already cover most of what the team needs. Adding another platform simply because it also has projects, agents and cloud execution would not make much sense.

Omnigent is easier to understand if we look at a team that uses several agent tools during the same development process.

A task does not always stay inside one agent

Take a developer who spends most of the day in Cursor but also uses Claude Code for a large refactoring task. Codex may be used to review the change. The company may have an internal agent for deployment, security checks or another workflow that is specific to the organization.

That is not an unusual setup anymore.

Each tool has its own session and its own view of the work. Claude Code knows what happened in the Claude session. Cursor has its project context. Codex has another conversation. An internal agent may have a completely separate configuration.

The source code moves easily through Git. The rest of the working history does not move quite as naturally.

During a long coding session, an agent may inspect many files, try several approaches, run tests, encounter errors and make decisions that are never captured in the final commit. A second agent looking only at the branch sees the result, but not necessarily everything that led to it.

Omnigent keeps a session around the work and allows different supported harnesses to participate in that session.

A task can begin with Claude Code and later be forked for Codex. Existing Claude Code and Codex CLI sessions can be imported. Projects can carry workspace and Git settings into new sessions. Sessions can also be shared or moved between hosts.

The practical benefit is fairly simple. The task has a history that can survive the move from one agent to another.

Several agents can work on the same engineering process

Omnigent also provides a way to organize work among several agents.

Polly, one of the examples included with the project, gives a good idea of how that can work. It coordinates a coding assignment, sends implementation work to agents running in separate Git worktrees and sends the resulting changes to another agent for review. The developer still decides whether the code is merged.

The development process itself remains familiar. There are separate branches, code changes, review, a pull request and a human decision at the end.

The agents involved in those steps do not all have to come from the same product.

One agent can use Claude Code. Another can use Codex. Other supported harnesses can participate as well.

Cursor can also coordinate many Cursor agents, and Claude can delegate work among Claude agents. Omnigent is aimed at a setup where the team has chosen more than one agent environment and wants to use them together.

For some organizations, that may reflect how developers are working anyway. Different teams have different preferences, and certain agents may perform better on particular kinds of work.

One place for policies around agent sessions

The policy system is another part of Omnigent that deserves attention.

Policies can be defined across the Omnigent server, attached to a particular agent or applied to an individual session. A rule can allow an action, ask for approval or block the action.

The built-in examples cover controls that engineering teams are likely to recognize immediately: spending limits, GitHub repository and branch restrictions, limits on tool calls, sandbox requirements and controls on the number of subagents.

There are also policies that watch agent behavior.

Loop detection can identify an agent repeatedly making the same call. Thrashing detection can identify repeated failures. Cost policies can keep track of spending over the life of the session.

Other policies keep information from earlier activity in the session. A running risk score can increase after particular actions. Information-flow rules can take into account whether confidential content has been accessed before allowing later writes.

That makes the session more than a container for conversation history. It also carries information that can influence how the agent is allowed to behave.

OpenAI, Anthropic and Cursor have their own security and administrative controls, and those remain part of their platforms. Omnigent provides an additional policy layer for the work being coordinated through different harnesses.

For a large organization using several coding agents, managing some of those rules in one place can be easier than reproducing them separately around every tool.

The execution environment can also be managed centrally

Coding agents need access to real systems. They edit files, run shell commands, install packages, call APIs and work with repositories.

Omnigent can run a session on a developer machine or inside an isolated environment.

The open-source project supports local isolation and several remote sandbox options, including Kubernetes-based execution. Managed Omnigent can also use Databricks Sandbox in supported environments.

That gives teams another choice for longer-running jobs. The work does not have to remain tied to the laptop where the session was started.

A repository task could continue in a sandbox while the developer moves on to something else. A scheduled task could run in the same type of controlled environment. Subagents could also be given isolated execution rather than unrestricted access to a developer machine.

For Databricks customers, the managed version connects this with the Databricks workspace. Workspace identity is used for access, and model traffic can be routed through Unity Gateway.

Choosing the agent as well as the model

Most organizations working with several foundation models are familiar with model routing.

A simple task may go to a lower-cost model. A more demanding task may be sent to a stronger model.

Omnigent adds the coding harness to that decision.

With Smart Routing, a task can be assigned to a model and an agent harness. Subagents can be routed in the same way.

A team might end up with combinations such as:

Repository task

Claude Code + Claude
Codex + GPT
another approved harness + another approved model

The choices can then be evaluated over time using actual work.

Omnigent tracks usage by session, model and harness. That allows teams to look at more than the cost of the individual model request.

For coding agents, I would want to know how much it cost to get a change into an acceptable state. That includes retries, tool calls, subagents, failed attempts and the amount of developer correction required before the pull request is ready.

A model with a lower token price does not automatically produce the lower-cost engineering task.

How I would place Omnigent

I would not replace ChatGPT Work, Claude Cowork or Cursor Projects simply to move everything into Omnigent.

They are different working environments and each has its own strengths.

A person doing research, writing a document or working with business applications may be perfectly comfortable in ChatGPT Work or Claude Cowork. A development team that has standardized on Cursor may prefer to keep its work inside Cursor Projects.

Omnigent makes more sense for an engineering organization that is already using several agent environments.

The same task may pass through Claude Code, Codex, Cursor and an internal agent. The organization may want common rules around those sessions, a controlled place for the agents to execute, visibility into cost and the ability to route work to different agent and model combinations.

Omnigent gives that work a common place to be managed.

That is also why I would pay less attention to whether its interface resembles ChatGPT, Claude or Cursor. Similar interfaces are becoming common across AI products.

The more interesting question is what happens to the work after it leaves one agent.

Can another agent continue with the history? Can the same policy still apply? Can the execution environment stay controlled? Can a second agent review the first agent's work? Can the organization see what all of this is costing?

For a team using one agent platform, these questions may be handled inside that platform.

For a team using several, Omnigent offers a way to manage them together without having to build all of that surrounding integration from scratch.

Nexa: Genie is a Minute Away

ravikr1 — Thu, 10 Sep 2026 17:50:27 GMT

Building an Explainable Semantic Intelligence Platform on the Databricks Lakehouse

We are building Nexa, a semantic intelligence platform for Databricks that turns a raw Lakehouse into a continuously maintained, explainable system that can compile natural language requests into governed Genie Agents. We are about 60 percent through implementation and want to share the architecture, the reasoning behind it, and why we think it solves a problem most catalog and semantic layer tools have not addressed properly: trust.

The problem we set out to solve

Every enterprise Lakehouse eventually accumulates the same gap. Unity Catalog tells you what tables and columns exist. Business users describe their world in a completely different vocabulary: revenue, shrink, available inventory, active customers. Someone, usually a data engineer, sits in the middle translating between the two, by hand, every single time a new question comes in.

Genie Agents solve part of this by letting business users ask questions in plain language. But a Genie Agent is only as good as the semantic mapping behind it, and most tools that generate that mapping treat the language model as the source of truth. That is a mistake. An LLM can guess that a column named inventory_qty probably means Available Inventory, but a guess is not a governed enterprise definition, and a wrong guess embedded silently in a Genie Agent produces confidently wrong answers that nobody catches until a business decision goes sideways.

Nexa is built on a different premise. AI can interpret technical truth, but it must never become the source of technical truth. Every semantic interpretation has to be traceable back to technical evidence, and every important AI decision has to be able to answer the question why do you believe this.

Two layers, kept honest

Nexa is architected as two distinct layers that are never allowed to merge into one blurry system.

The first is the Technical Knowledge Graph. This is the canonical, machine derived representation of the actual Databricks environment: schemas, tables, columns, primary and foreign key relationships, lineage, query behavior, data quality signals, tags, and physical statistics. Nothing in this layer is invented. It comes directly from Unity Catalog metadata, information schema, system tables, and observed lineage.

The second is the Enterprise Semantic Layer. This is the governed interpretation on top: business concepts, metrics, KPIs, dimensions, synonyms, and business rules. This layer is derived, not authoritative on its own. Every mapping from a business concept to a technical asset carries evidence, and that evidence is inspectable.

We deliberately separate two kinds of explainability that most platforms conflate. A Graph ML model answers why we believe two technical entities are related. A language model answers why we interpret a technical entity as a particular business concept. Keeping these two questions and their answers separate means that when something goes wrong, you know immediately whether the problem is structural or semantic, instead of debugging a black box.

How the pieces fit together

The system is organized into five planes.

The experience plane is a React application covering the Ontology Explorer, Semantic Explorer, graph visualization, Genie Builder, Genie Chat, governance review, explainability views, impact analysis, and drift and quality monitoring.

The application plane is a Node.js orchestrator handling authentication, the Databricks API client, the Genie Agent API client, job orchestration, semantic compiler orchestration, session management, and streaming.

Below that sit two engines working in parallel. The Python AI Engine handles profiling, embeddings, GraphSAGE and GAT based edge scoring, entity resolution, semantic matching, model evaluation, and MLflow tracking. The Semantic Compiler handles intent interpretation, concept resolution, metric selection, join planning, Genie configuration, instruction generation, benchmark generation, and quality gates.

Both feed into the Semantic Intelligence Plane, which holds the Technical Knowledge Graph, the Enterprise Semantic Layer, an Evidence Engine, a Trust Engine, an Ontology Critic, and the governance and human feedback loop.

At the bottom is the Data Truth Plane: Unity Catalog for information schema, table and column metadata, lineage, and tags, Delta Lake for ontology state, profiles, evidence, feedback, semantic definitions, and evaluation results, and Databricks system tables for operational signal.

We favor native Databricks capabilities wherever they provide reliable functionality. Unity Catalog metadata, lineage, and system tables get you most of the way to a technical graph before you need to invoke a single model. Graph ML gets reserved specifically for ambiguous relationship discovery, not as a default step, because deterministic signals are cheaper, faster, and more trustworthy than probabilistic ones whenever they are available.

What makes disagreement a feature, not a bug

One design decision we expect to be controversial in a good way: when Graph ML, the language model, lineage, and business rules disagree with each other, Nexa surfaces the conflict instead of picking a winner silently.

Here is a real pattern we designed around. Say a column inventory_qty gets mapped to a business concept. The graph confidence is 97 percent that this is a valid technical relationship. The LLM's semantic confidence is 84 percent that the mapping name is correct. But the same field has historically been used as both Available Inventory and On Hand Inventory in existing definitions elsewhere in the enterprise. Nexa detects this and marks it as requiring human review rather than auto approving it. We consider that outcome a success, not a failure. A platform that hides this kind of ambiguity behind a high confidence score is more dangerous than one that has no opinion at all.

This is enforced through what we call the fail closed principle. Low confidence or conflicting semantic decisions require human review rather than automatically becoming trusted enterprise definitions. High risk changes never sail through on autopilot.

Semantic Diff and Drift Timeline

Governance logs traditionally tell you that a decision happened. We wanted something better: a browsable history of what a concept has meant over time.

Every mapping between a business concept and a technical asset is stored as an append only version in a Delta table, never overwritten in place. Each version carries its graph confidence, its LLM confidence, its evidence reference, who approved it, and why. When two approved mappings for the same concept conflict, that gets logged separately with its own resolution workflow.

In the Semantic Explorer, this becomes a Concept Timeline: a horizontal sequence of every version a business concept has ever had, color coded by status, with a compare mode that shows exactly what changed between two versions and which Genie Agents would be affected by that change. If someone asks why the definition of Available Inventory changed last quarter, the answer is a few clicks away instead of a Slack archaeology project.

Counterfactual Genie: simulating impact before you commit

This is the feature we are most excited about, because we have not seen it anywhere else in the catalog or semantic layer space.

Before a semantic mapping, a schema change, or a confidence threshold adjustment gets approved, Nexa can simulate its effect through the graph and show exactly which live Genie Agents, dashboards, and metrics would change, and by how much, without ever touching the real system.

The mechanism is a shadow overlay rather than a shadow environment. We do not stand up a second graph or a second Genie deployment. Instead, a small hypothetical diff object gets passed as an optional parameter into the same graph read functions and the same semantic compiler code path that runs in production. When the overlay is present, reads get patched in memory before being returned. No write ever touches the real graph or the real Delta tables during a simulation. This guarantees the simulation can never drift from real behavior, because it is running the exact same logic, just against a hypothetical state.

In practice this means a reviewer looking at a proposed mapping change can click Simulate Impact and see, within the same review screen, a checklist of every Genie Agent that references the affected concept, whether each one would materially change, and a side by side diff of the generated query configuration before and after. If lowering an auto trust threshold would cause 23 additional mappings to auto approve last quarter, and 3 of them would contradict each other, that shows up before the threshold change is committed, not after.

We think of this as extending our fail closed principle into fail closed with a preview. Governance is not just a gate, it is a gate with a window.

Why this matters for enterprise

A business user should eventually be able to say build me a supply chain intelligence agent and have the platform work out what they mean, which business concepts and metrics are involved, which technical assets support those concepts, which relationships are trustworthy enough to use, what data should be exposed, how the Genie Agent should be configured, whether the result passes quality checks, and why every one of those decisions was made.

That last part, the why, is the piece most platforms skip. Nexa treats explainability and governance as core infrastructure rather than an afterthought layered on top of a text to SQL engine. The technical graph tells you what exists. The semantic layer tells you what it means. The evidence engine tells you why you should believe it. The governance layer tells you whether it is trusted. The semantic compiler turns all of that into a working Genie Agent, and the counterfactual simulator lets you see the consequences before you sign off on anything.

Where we are

We are roughly 60 percent through building Nexa end to end: the Technical Knowledge Graph ingestion from Unity Catalog, the Evidence Engine, and the core Semantic Compiler path are the furthest along, with the Semantic Diff timeline and Counterfactual Genie simulator actively being built out next.

Genie is a minute away. We built Nexa to make sure that minute is trustworthy, not just fast.

If you are working on similar problems around semantic layers, Genie Agent governance, or explainable AI on the Lakehouse, we would love to compare notes.

Smart Routing in Unity AI Gateway: 30%+ Cost Savings on Coding Tasks

Tushar_Parekar — Thu, 10 Sep 2026 17:28:57 GMT

Coding teams now have more models and harnesses to choose from than ever. Smart Routing in Unity AI Gateway helps remove that choice overload by matching each coding task to a model, and, with Omnigent, a harness that fits its complexity and cost.

Key highlights

Route by task, not by default: Smart Routing classifies a task using signals from the initial prompt, then selects a model suited to the work. Simpler tasks can use lower-cost models, while complex tasks can be escalated to more capable options.
Preserve cache efficiency: The router uses task-aware routing at the start of a session rather than switching models on every request. Keeping a session on the selected model helps protect cache-hit rates and control end-to-end task cost.
Reduce cost without blunt caps: Internal results showed about 35% savings, while public coding benchmarks showed 56% cost savings. The goal is not to minimize spend at any cost, but to improve productive output per dollar while preserving quality.
Route across models and harnesses: Smart Routing works with Claude Code and Codex through Unity AI Gateway. With Omnigent, teams can select both the model and coding harness, including for sub-agents.
Measure productivity, not just price: Coding-session traces provide the feedback loop for evaluating model mix, routed sessions, dollar savings, and developer experience. Routing should be refined using real workloads rather than cost alone.

👉 Read the full post here

Databricks Community Contest | Winners of the Genie-Powered App Challenge!

Tushar_Parekar — Thu, 10 Sep 2026 16:12:10 GMT

✨ 🌟 💫

✨ The Genie has spoken

And the winners of the Genie-Powered App Challenge are… 🎉

A few weeks ago, we threw down a challenge: build something incredible with Databricks Apps and Genie Agent. And wow, you delivered. 🔥

From smart, real-world problem-solvers to wildly creative passion projects, we were blown away by the imagination, skill, and sheer variety of what you built across two tracks. Picking winners was not easy, but the scores are in, and it’s time to celebrate! 🥳

👏 With thanks

First, a massive thank you to our judges.

Every single submission was reviewed from multiple angles, against the criteria we shared at launch. A big effort behind the scenes, thank you to the brilliant panel who made it happen:

Ashwin

@Ashwin_DSA

💜 Community Fellow

Emma

@emma_s

💜 Community Fellow

Anuj

@anuj_lathi

💜 Community Fellow

Philip Laserstein

@PL_db

🧞 Genie Community Team

Ashwin, Emma & Anuj · our Community Fellows 💜
The awesome Bricksters who show up: in the community, for customers, and for you. Each is an expert in their own domain, which made them the perfect crew to judge your work from every angle.

Philip · from the Genie Community team 🧞
Who joined the panel to judge the apps through a true Genie lens.

✨ 🧲 ✨

🏆 Wishes granted

And the scores are in.

🎯 Track A

Real-World Problem-Solving

35.7

POINTS

🥇 1st Place · Gold

ChicagoPulse

Brian Denis Castelino

@bcastelino 

34.7

POINTS

🥈 2nd Place · Silver

Bushfire Exposure Across Victoria’s Powerline Network

Vivekanandan Priya Kumar

@VivekKumar 

🎨 Track B

Creativity & Originality

POINTS

🥇 1st Place · Gold

Le Greffier

Ust Oldfield

@UstOldfield 

34.7

POINTS

🥈 2nd Place · Silver

Prove It

Ivan Vydrin

@ivanvyd 

🎁 Winners, swag is heading your way! Our team will reach out soon to grab your details.

🎤 On stage

Taking the stage!

The top project from each track has earned something special: a spot to present their work live in an upcoming BrickTalk session.

✨ Track A Spotlight

ChicagoPulse

Brian Denis Castelino

@bcastelino

✨ Track B Spotlight

Le Greffier

Ust Oldfield

@UstOldfield

We’ll be reaching out to our top two soon to finalize the details, stay tuned for when and where you’ll be able to catch these projects live!

👏 So close

Right on the winners’ heels.

Two projects absolutely deserve a shoutout:

→ The Exfiltration · Oleksandr Kinashchuk (@okinashchuk)

→ Gridiron Genie: an NFL matchup room powered by Databricks Genie Agent · Dan Wiltse (@dan_wiltse )

Incredible work, you made the final calls genuinely tough. 💪

💜

To every participant

You pushed what’s possible with Genie Agent.

You didn’t just enter a challenge, you pushed what’s possible with Genie Agent and shared it with the whole community. That’s a big deal.

🏅 A participation badge will soon appear on your profile page.

📝 We’ll also be sharing the judges’ feedback with you, so you can see what stood out and keep leveling up.

✨ 🧞 ✨

👋 What’s next

Keep an eye out for the next mystery challenge. 🤫

It’s been genuinely lovely watching your ideas come to life. The community frequently hosts challenges like this, so keep an eye on the community for what’s coming.

See you around the community! 💙

Materialized Views vs Streaming Tables in Databricks: A Practical Guide for Data Engineers

Islam_hoti — Thu, 10 Sep 2026 11:18:56 GMT

As data platforms move toward more declarative and incremental processing, one question comes up frequently when building pipelines in Databricks:

Should I use a Materialized View or a Streaming Table?

Both are important building blocks in Lakeflow pipelines, but they solve different problems.

Materialized Views

A Materialized View stores the precomputed results of a query rather than calculating the result every time someone queries it.

When the upstream data changes, the Materialized View can be refreshed so that the stored result reflects the latest data.

This makes Materialized Views especially useful for transformations such as:

Aggregations

Data cleaning

Joins

Business level transformations

BI reporting tables

Dashboard datasets

Databricks can also incrementally refresh Materialized Views when the query and underlying data support it, which can avoid recomputing the entire dataset. (Databricks Documentation)

A simple example:

CREATE OR REFRESH MATERIALIZED VIEW daily_sales

SELECT

order_date,

SUM(amount) AS total_sales

FROM sales

GROUP BY order_date;

Instead of recalculating the aggregation every time a user queries daily_sales, Databricks maintains the result for us.

Materialized Views can also be refreshed manually, on a schedule, or automatically based on upstream updates. (Databricks Documentation)

Streaming Tables

Streaming Tables are designed for data that continuously arrives.

Typical examples include:

Application events

IoT data

Kafka or Event Hub streams

Cloud file ingestion

Transaction events

Operational logs

In a Lakeflow pipeline, a Streaming Table can be created with SQL:

CREATE OR REFRESH STREAMING TABLE bronze_orders

SELECT *

FROM STREAM read_files(

'/Volumes/catalog/schema/raw/orders',

format => 'json'

);

The important difference is that the source is processed using streaming semantics.

Instead of processing the complete dataset during every pipeline update, the pipeline processes newly arriving data incrementally.

Databricks uses the STREAM keyword when a source should be read using streaming semantics. (Databricks Documentation)

A Simple Architecture

A common architecture can combine both concepts.

Source Data

↓

Streaming Table

↓

Bronze

↓

Streaming / Transformation Layer

↓

Silver

↓

Materialized View

↓

Gold / BI / Analytics

For example, incoming orders can first land in a Streaming Table.

The data can then be cleaned and validated in the Silver layer.

Finally, a Materialized View can aggregate the data for dashboards, reporting, or analytics.

This allows streaming ingestion and analytical transformations to work together instead of treating batch and streaming as completely separate architectures.

When Should You Use Each One?

Use a Streaming Table when the main requirement is continuously processing newly arriving data.

Use a Materialized View when the main requirement is maintaining the result of a transformation or analytical query.

A simple rule I use is:

Streaming Table = how data arrives and moves

Materialized View = how transformed results are maintained

There are exceptions, but this distinction makes architecture decisions much easier.

One Important Streaming Consideration

Streaming semantics work best with append oriented sources.

If existing records in the streaming source are modified or deleted, the pipeline needs an appropriate strategy for handling those changes.

Databricks provides options such as SKIPCHANGECOMMITS for specific scenarios, while CDC workloads can use dedicated change data capture patterns. (Databricks Documentation)

Understanding this behavior is important before simply converting every pipeline into streaming.

Final Thoughts

Materialized Views and Streaming Tables are not competitors.

They are complementary components.

A well designed Databricks architecture can use Streaming Tables to efficiently ingest and incrementally process incoming data while using Materialized Views to maintain cleaned, aggregated, and analytics ready datasets.

The key is to choose the object based on the behavior the workload requires rather than simply choosing between “batch” and “streaming.”

For Data Engineers, understanding this distinction becomes increasingly important as Databricks continues moving toward declarative pipeline development with Lakeflow and Spark Declarative Pipelines. (Databricks Documentation)

What patterns are you currently using for Materialized Views and Streaming Tables in your Databricks pipelines?

Announcement | How we eliminated $1 million a year of wasted AI agent spend in one hour

Tushar_Parekar — Thu, 10 Sep 2026 10:46:07 GMT

Silent MCP tool failures can make agents look productive while quietly increasing token usage and wait time. By combining Unity Gateway tracing with Genie One, Databricks engineers turned that hidden waste into a ranked, fixable bug list, identifying an estimated $1.2 million per year in wasted spend and lost productivity.

Key highlights

Tool failures create hidden costs: Across one agent fleet, seven MCP-server bugs produced 1,409 errors per day, an estimated $499K in annual token costs, and about 12,000 hours of annual agent wait time.
Tracing makes agent behavior observable: Unity Gateway automatically records OpenTelemetry traces for MCP tool calls, including tool names, arguments, errors, token counts, latency, and session IDs. The unified trace table makes it possible to connect wasted spend to specific tools and failures.
Genie One turns traces into answers: Instead of spending hours exploring schemas and writing SQL, the team asked natural-language questions about recurring errors, recovery time, token cost, and wait time. Genie One returned a prioritized list of what to fix.
Design tools for how models actually call them: LLMs make reasonable guesses when tool signatures are ambiguous. Tools should handle common variations gracefully, provide useful errors, and avoid crashing on inputs that are semantically valid but shaped differently than expected.
The fix loop can be fast: Once the highest-impact failures and their causes were clear, coding agents helped apply the fixes. The full cycle, find, quantify, and fix, took about an hour.

👉 Read the full post here 👈

CUSTOMER STORY | Rippling powers AI-driven GTM with Genie Agents on Databricks

Tushar_Parekar — Wed, 09 Sep 2026 14:59:21 GMT

“Databricks gives us a scalable foundation for agentic GTM with Genie. We can combine governed data, machine learning and semantic retrieval to build new experiences quickly.” - John Kutay, Head of Growth Engineering, Rippling

Rippling’s go-to-market teams need timely, trusted data to personalize outreach, prioritize accounts and act on new opportunities. By building GrowthOS on Databricks and integrating Genie Agents, Rippling created a governed conversational layer where employees and AI agents can work from the same real-time GTM intelligence.

Key highlights:

2,800+ operators use Genie-powered agents monthly, generating more than 2 million AI-powered queries across sales, marketing, operations and analytics.
33% lift in demos booked during staged A/B tests of AI-driven personalization.
20% lift in new opportunities generated through data-driven GTM intelligence.
A unified foundation for people and agents: Rippling uses Spark Declarative Pipelines, Delta Lake and machine learning pipelines to continuously process first- and third-party data.
Trusted, reusable intelligence: Entity resolution reconciles customer and prospect records across hundreds of millions of records, while AI Search supports grounded semantic retrieval with lower inference costs and faster responses.

🔗 Check out the full story 👈

Learn Databricks Lakeflow | Ingest, Orchestrate, and Build pipelines on one platform.

Tushar_Parekar — Wed, 09 Sep 2026 10:00:13 GMT

Databricks Training & Certifications

Learn Databricks Lakeflow

Unified data engineering: ingest, orchestrate, and build pipelines on one platform.

🔌 Ingest ⚙️ Orchestrate 🎨 No-code ETL

Data engineering on Databricks comes together under Lakeflow. Bring data in from almost anywhere with Lakeflow Connect, turn workloads into reliable production jobs with Lakeflow Jobs, and build ETL pipelines visually with Lakeflow Designer, all on the Databricks Data Intelligence Platform.

The catalog covers three Lakeflow courses on the Data Engineer track, two of them with a free self-paced option. Follow the flow below. 👇

1 Ingest the data · Lakeflow Connect

Data Engineer · Associate

Data Ingestion with Lakeflow Connect

A comprehensive introduction to Lakeflow Connect for ingesting data into Databricks at scale. Work with Standard and Managed connectors and with batch, incremental batch, and streaming ingestion. Load from cloud object storage using CTAS, COPY INTO, and Auto Loader, handle mismatched records with the rescued data column, flatten semi-structured JSON, and bring in data from databases and SaaS apps. Rounds out with Partner Connect, MERGE INTO, and the Databricks Marketplace, all on UC tables and the medallion architecture.

Free · Self-paced · 2H → Paid / Subscription · Lab · 3H → Paid · Instructor-led · 4H →

2 Orchestrate the workload · Lakeflow Jobs

Data Engineer · Associate

Deploy Workloads with Lakeflow Jobs

Orchestrate and automate data, analytics, and AI workflows with Lakeflow Jobs as a unified orchestration platform. Design workloads as Directed Acyclic Graphs (DAGs), configure scheduling, and use advanced features such as conditional task execution, run-if dependencies, and for-each loops. Covers best practices for production-ready pipelines: compute selection, modular orchestration, error handling, and fault-tolerant design.

Free · Self-paced · 2H → Paid / Subscription · Lab · 3H → Paid · Instructor-led · 4H →

Instructor-led available in English, 日本語, Português BR, 한국어, Español, and française.

3 Build it without code · Lakeflow Designer

Go further: build a full medallion pipeline on a visual canvas, with an AI assistant writing the transformations for you. Paid / subscription lab.

Data Engineer · Professional

No-Code ETL with Lakeflow Designer

Build a complete medallion ETL workflow, bronze to silver to gold, visually and without writing pipeline code. Assemble pipelines on a visual canvas using operators, and use the built-in AI authoring assistant, Genie Code, to generate operators from natural language. Clean and reshape data into silver, aggregate and join into gold, schedule the workflow, then deliver insights through an AI/BI dashboard, a Genie Space, and Genie One. An optional bonus uses the AI Function operator to derive sentiment from free-form feedback.

Paid / Subscription · Lab · 2H →

Browse all Lakeflow courses in the catalog →

Ingest. Orchestrate. Build. One data engineering experience on the Databricks Data Intelligence Platform.

CosmosGenie — Your Universe, Answered (Genie-Powered App Challenge )

sudiptob-DA — Tue, 08 Sep 2026 20:00:31 GMT

My son asked me one evening whether any asteroids were going to hit Earth. The
data to answer him exists — NASA publishes it daily, for free — but it lives in
JSON behind API keys, in units like astronomical units and X-ray flux classes,
built for people who already know what they're looking for. A curious ten-year-old
is not that person.

CosmosGenie fixes the interface, not the data. Ask anything about space in plain
English — asteroids, eclipses, the Moon, planetary line-ups, rocket launches —
and it queries live NASA, USNO and JPL data and answers in a sentence.

**How it's built.**

Eight free public APIs feed a full medallion architecture on
Databricks Free Edition. Two ingestion paths — standalone notebooks on a Lakeflow
Job, and a Lakeflow Spark Declarative Pipeline with bronze → silver → gold and
data-quality expectations — land eight silver tables (the source of record) and
four gold tables tuned for Genie.

[ Free Public APIs ] [ Databricks Free Edition ]

NASA NeoWs ─────┐ Lakeflow SDP Pipeline
NASA DONKI ─────┤ ┌─ bronze/ (raw API pull)
├──► SDP Pipeline ────►├─ silver/ (clean + DQ expectations)
│ └─ gold/ (business logic, KPIs)
USNO Moon ─────┐
JPL / Curated ───┤
NASA Eclipse ────┼──► Lakeflow Job ────► Delta Tables (cosmos.space.*)
The Space Devs ──┤
Spaceflight News ┘
│
┌─────────┴──────────┐
Genie Space
└─────────┬──────────┘
Streamlit App

**The app** is Streamlit on Databricks Apps with the Genie Agent attached as a
resource:

An aurora theme, a live KPI bar, an interactive 12-month events timeline
where clicking an event asks a question, and threaded chat that returns prose, the
generated SQL, a table and a tailored visual.

**Why Genie.**

Remove it and what's left is four numbers and a timeline — no
dashboard, no filter panel, no pre-built report. Every ranking and caveat is
generated live from a question nobody wrote in advance. Almost all the effort went
into the semantic layer: column comments, instructions and sample questions.

What I learned.

The natural-language part is only as good as the semantic layer behind it.The payoff is that once Genie understands the data this well, it reliably handles questions I never anticipated and never wrote an example for. It even explains its own reasoning. The lesson: the model isn't the hard part — describing your data clearly is. Invest there, and the natural-language experience takes care of itself.

Expanding Genie Agents: Deep analysis, file reasoning, and more

Tushar_Parekar — Tue, 08 Sep 2026 15:19:07 GMT

Genie Agents are becoming more powerful and easier to curate, helping teams move from quick data lookups to deeper analysis, richer context, and shared decision-making.

Key highlights

Go deeper with Agent mode and APIs: Agent mode supports multi-step research: it can develop a plan, iterate across queries, and return findings with visualizations and citations. Agent mode APIs bring this experience to custom applications, chatbots, scheduled reports, and internal tools, with streaming responses, follow-up conversations, and programmatic access to messages and attachments.
Reason across tables and files: Genie Agents can analyze PDFs, documents, slide decks, and images stored in Unity Catalog volumes alongside structured data. Builders can attach up to 10 volumes, while Unity Catalog permissions and optional content search help keep retrieval governed and efficient.
Curate agents with Genie Code: Genie Code can create a baseline agent, help diagnose conversation or benchmark failures, summarize user feedback, and suggest improvements to instructions, example SQL, or knowledge-store configuration. Authors review suggestions before saving them.
Share conversations and insights: Users can share conversations with teammates and agent authors. Shared links stay current as new messages, edited visualizations, and follow-up analyses are added.

Together, these updates help teams build Genie Agents that can investigate more nuanced questions, combine business data with document context, and improve through real usage and feedback.

🔗 Read the full post here

🌟 Community Pulse: Your Weekly Roundup! August 31 – September 06, 2026

Advika — Tue, 08 Sep 2026 12:23:53 GMT

The Weekly Digest • August 31 – September 6

Community Pulse

Your weekly pulse: new articles, real answers, and the discussions worth watching. 📨 📌

🏅 This Week's Community Stars

This week's difference-makers – and a shout-out to the new members earning their spot here 👏:

@balajij8

@Satyasai

@data_pulse

@srini_ve

+ Top Brickster contributor:

@Ashwin_DSA

Note: Names are listed in no particular order.

📖 Knowledge Hub

✍️ Community Articles

📌 Technical Blogs

Technical Blog Lakeflow Connect: Message Bus Ingestion - Now shipping your logs directly! (Beta)Read article →	Technical Blog GeoGenie 2: Ask your map a question in plain English Read article →
Technical Blog Custom Visualizations in Databricks AI/BI Dashboards: Build a Dumbbell Chart Read article →	Technical Blog Managing Databricks Genie Agents as Code with Databricks Declarative Automation Bundles (DAB)Read article →

Active Discussions

Threads that kept the community talking this week 👀

🔥

Streaming tables fail with DELTA_STREAMING_INCOMPATIBLE_SCHEMA_CHANGE_USE_SCHEMA_LOG

↗

🔥

Databricks Data Ingestion

↗

🔥

What is the difference between a managed table and an external table in Databricks?

↗

🔥

Lakaflow connect Data Ingestion gatway pipeline fail often and restarts automatically

↗

🔥

create_auto_cdc_from_snapshot_flow Python session resolution fails if having multiple snapshot flows

↗

🔥

VS Code and Connect - Manual Environment Setup

↗

These are just the highlights — head to the community to see everything that's buzzing.

🔥 Genie-Powered App Showcase

From idea to app – see the Genie-powered projects our community built during this period ✨:

Genie App Genie Powered App Challenge Explore →	Genie App A Conversational Trade Promotion Optimization App with Genie at the Core Explore →	Genie App AegisBank: Autonomous Fraud Ring & Mule Surveillance Copilot (Track A)Explore →
Genie App Ledgerly: A Genie-Powered Business Health Copilot for Small Businesses Explore →	Genie App LakeOps - A Lakehouse Operation Observability App Explore →	Genie App Genie SQL Quest: A Genie-Powered Arcade for Learning SQL Explore →
Genie App OpsPulse- From Operational Signals to Data-Backed Answers with AIBI Genie Explore →	Genie App Crux — Construction Intelligence Powered by Databricks Genie Explore →	Genie App BI Rationalization Genie: Turning Report Sprawl into Conversational Decisions with Databricks Genie Explore →
Genie App MAD DATA LAB: Wonderful. Something Is Wrong.Explore →	Genie App SleepLens: Turning Multimodal Sleep Data Into Conversations With Databricks Genie Explore →	Genie App Insurance Intelligence Copilot – Powered by Databricks Genie Explore →
Genie App MuleGraph Investigator: Uncovering Money Mule Networks with Databricks Genie Explore →	Genie App CurePath: fewer avoidable repossessions, found by asking questions Explore →	Genie App ChicagoPulse: Ask Your City What’s Changing and Why Explore →
Genie App Policy Time Machine: Teaching Genie to Answer "What Changed Before the Claim?Explore →	Genie App Research It!Explore →	Genie App Crick Genie XI Explore →

🎙 Community Newsroom: Announcements & Events

● World tour

DATA + AI World Tour 2026

● world tour

The Data + AI World Tour lands Sept 16 - apply before seats are gone

● Customer Story

CUSTOMER STORY | PAR Technology builds modern AI-powered intelligence on Databricks Genie

● Customer Story

CUSTOMER STORY | Kraken governs utility data at scale with Unity Catalog

● Customer Story

CUSTOMER STORY | Brasilprev turns scattered knowledge into governed insights with Genie

✅ Questions That Found Their Answer

A few of the threads that wrapped up with a solution this period:

✔

External location not accessble with job and general purpose cluster but works fine with serverless

↗

✔

How can i rename a column in a delta table?

↗

✔

Difference between Workspace and Unity Catalog experiments when using MLflow autologging?

↗

✔

Thoughts on Using Remix for Data-Focused Applications

↗

✔

Is it possible to embed a Genie Space inside a Power BI report?

↗

These are just the highlights — head to the community to see everything that's buzzing.

Your turn!👀
Saw a question you can answer? A story worth sharing? Every great thread starts with someone hitting "reply".
Go hit reply 😎👋.

Solution Accelerator Series | Simplifying Product Onboarding With Generative AI

Tushar_Parekar — Tue, 08 Sep 2026 10:33:53 GMT

Product onboarding can be complex and time-consuming when teams work with inconsistent data formats, inaccurate information and multiple stakeholders. The Simplifying Product Onboarding With Generative AI Solution Accelerator shows how generative AI can help automate product data creation and review while supporting faster onboarding and improved data accuracy and quality.

With this Accelerator, you get

Ready-to-use resources: pre-built code, sample data and step-by-step instructions ready to go in a Databricks notebook
Automate product data creation and review: use generative AI to simplify key onboarding tasks.
Improve data quality: support more accurate and consistent product information.
Accelerate onboarding: help teams move through the process more efficiently.
Scale with product portfolios: support onboarding as the number of products grows.

🔗 Launch Solution Accelerator 👈

Databricks Lakehouse Industry Data Models: What Data Engineers Can Learn from the GitHub Repository

AmitDECopilot — Tue, 08 Sep 2026 05:37:56 GMT

I recently explored the Lakehouse Industry Data Models repository published under Databricks Industry Solutions on GitHub.

The scale of the repository is impressive:

• 40 industries
• 80 models across ECM and MVM variants
• More than 23,000 tables and data products
• More than 156,000 foreign-key relationships
• More than 11,000 metric views

Each industry provides two model options:

• Expanded Coverage Model for broader domain coverage
• Minimum Viable Model for a smaller, implementation-focused starting point

The repository includes much more than entity names. Engineers can inspect model JSON, SQL schemas, relationships, metric views, ontology tags, generated documentation, and DBML diagrams.

It also provides tooling to install a selected model into Unity Catalog, populate it with referentially consistent sample data, and visually explore relationships through the model viewer.

My main takeaway is that these models are most valuable as governed starting points rather than final enterprise designs.

A team could use them to accelerate:

• Domain discovery
• Data-modeling workshops
• Data-product identification
• Source-to-target mapping
• Metric-view planning
• Governance and metadata discussions
• AI-assisted architecture experiments

The business definitions, grain, keys, regulatory requirements, and source-system realities still need validation by engineers and domain experts. AI can accelerate the initial structure, but production architecture still requires human review and contextual knowledge.

GitHub repository:
https://github.com/databricks-industry-solutions/lakehouse-industry-data-models

My detailed review:
https://dataengineeringcopilot.com/blog/databricks-industry-data-models-ai-assisted-architecture

I would be interested to know whether others are using these models for architecture discovery, prototypes, or production planning.

#Databricks #DataEngineering #Lakehouse #DataModeling #UnityCatalog #MetadataManagement

Stop Asking an LLM Judge Questions Your Code Can Answer

ivanvyd — Tue, 08 Sep 2026 04:02:53 GMT

Suppose a RAG application returns this:

{
  "response": "You can return the item within 90 days.",
  "citations": ["returns-policy"]
}

The cited document allows returns within 30 days.

A citation-presence check should pass this response. A check that the source supports the answer should fail it.

You can inspect the citations field in code. Assessing whether a free-form answer is supported by a document is a reasonable job for an LLM judge. Keep those scores separate so a present citation cannot be mistaken for a supported claim.

Define the requirement first

For this example, the requirement is:

Every response in this evaluation dataset must contain a non-empty list of nonblank citation IDs.

That gives us a structural check. Assessing whether every factual claim has a supporting citation would require identifying those claims and comparing them with the evidence.

MLflow supports @scorer functions that consume application outputs and return Boolean results. Here is the structural check, using the documented code-based scorer API:

from mlflow.genai.scorers import scorer


@scorer
def has_citation(outputs: object) -> bool:
    """Check for a non-empty list of nonblank citation IDs."""
    if not isinstance(outputs, dict):
        return False

    citations = outputs.get("citations")
    return (
        isinstance(citations, list)
        and len(citations) > 0
        and all(
            isinstance(citation_id, str) and citation_id.strip()
            for citation_id in citations
        )
    )

A length check alone would accept "returns-policy" or [None]. This version rejects both.

The function checks only the citation field. It does not validate the whole response schema, resolve the IDs, or read the documents. It also assumes citations are required for every record in this dataset. An application that permits uncited greetings or refusals needs a different contract.

With an MLflow tracking destination and experiment configured, you can evaluate three toy records:

import mlflow

data = [
    {
        "inputs": {"question": "What is the return window?"},
        "outputs": {
            "response": "You can return the item within 90 days.",
            "citations": citations,
        },
    }
    for citations in (["returns-policy"], [], "returns-policy")
]

results = mlflow.genai.evaluate(
    data=data,
    scorers=[has_citation],
)

MLflow accepts precomputed inputs and outputs, so this example does not need a predict_fn. See the mlflow.genai.evaluate API reference.

The predicate returns True, False, and False for those citation values. The incorrect 90-day claim passes through untouched because this scorer never examines it.

Check citation structure and support separately

For the opening example, I would keep three requirements separate:

Requirement	Evidence needed	Evaluation approach
Citation IDs have the required structure	Application output	Code
Each ID refers to a document retrieved for this request	Output and recorded retrieval results	Code
The answer’s claims are supported by the retrieved material	Answer and document content	Semantic evaluation

The second check should use IDs recorded by the retrieval layer. Comparing citations against another list generated by the model would leave both sides of the check dependent on its output.

The third requires reading the evidence. A document can exist and have been retrieved while still contradicting the answer.

MLflow’s RetrievalGroundedness judge assesses support from the supplied context. Its documented trace-based workflow requires at least one RETRIEVER span, with inputs and outputs on the root span. Adding it to the output-only example above would leave it without the required retrieval evidence. See the groundedness judge requirements.

Groundedness has a further limit: an answer can faithfully repeat an outdated or incorrect source. Evaluating support from that source does not independently verify that the source is true.

Check what the Python function actually does

MLflow uses code-based scorer for a Python-defined evaluator. That function can call an LLM, wrap a built-in judge, or run other custom logic. The custom-scorer documentation covers these options.

The @scorer decorator therefore tells you nothing about whether the result is deterministic.

For the checks discussed here, I mean explicit rules applied to fixed, recorded evidence without a model call. Schema validation, required fields, allowlists, argument constraints, and comparisons against known expected values fit that description.

Those rules still need to measure something useful. A regex that finds a URL may return the same result every time, but naming that result answer_verified would overstate what it establishes.

Tool names are only part of the evidence

Suppose a test case requires:

get_customer → issue_refund

The intended requirement might include ordering, matching customer IDs, and a successful transaction. A set-membership check establishes only that both tool names appear.

MLflow traces contain requests, responses, tool parameters, timing, and other execution data for instrumented spans. Use the fields needed by the requirement. A list of names cannot establish that the customer lookup finished before the refund started or that the refund used the correct customer ID.

To confirm the refund completed, inspect a trustworthy transaction result or the system that records it. A recorded invocation alone establishes an attempt.

Missing telemetry also needs care. An absent span might mean the tool never ran, or that instrumentation failed to capture it. Unless coverage is known to be complete, report missing evidence rather than silently returning a pass.

Now consider a broader question:

Given the conversation and refund policy, should the agent have issued a refund?

Some cases reduce to an explicit rule such as “refunds above this amount require approval.” That rule belongs in code. Other cases involve interpreting the conversation or an ambiguous policy exception, where an LLM assessment and human review may be appropriate.

Choose the evaluator based on the decision you need to check, rather than assigning all tool-use or policy questions to a judge.

Keep the individual scores

A single prompt that checks formatting, citations, tool use, groundedness, relevance, and policy compliance leaves several possible explanations for FAIL. Separate assessments make those failures easier to investigate.

For a RAG application, I would keep has_citation and citation-ID checks alongside RetrievalGroundedness. RelevanceToQuery assesses whether the response answers the request, while Guidelines can evaluate specified natural-language criteria. The built-in judge documentation describes those criteria and their requirements.

A release gate can combine the results while retaining each assessment for debugging. Record skipped checks separately, too. A skipped groundedness check supplies no evidence of a pass.

Before relying on a semantic judge to block a release, I would compare its decisions with human-reviewed examples. The choice of evaluator still needs validation against the cases it will judge.

Enforce rules before the action

Consider this requirement:

refund amount must be greater than zero

Validate it before executing the refund. A scorer can inspect recorded attempts afterward, but runtime validation has to happen on the path that executes the operation.

Databricks production monitoring evaluates a configurable sample of incoming traces and attaches assessments. The documentation describes it as a Beta feature. Those assessments can help identify failures in recorded traffic; they do not stop the operation before it happens.

Reusing development scorers in production also has practical requirements.

For Databricks-managed monitoring, custom scorers must use @scorer and be defined and registered from a Databricks notebook. Custom class-based Scorer subclasses are not supported. See the custom-scorer restrictions.

The functions must be self-contained, with required imports inside the function body. After registration, call .start() with a sampling configuration, as described in the monitoring setup.

Check the available data before reusing a scorer. Registered production scorers obtain inputs and outputs from traces; the Databricks scorer reference says expectations is unavailable. A scorer that relies on hand-labeled expected tools in an offline dataset will need changes before it can evaluate live traffic.

Support also differs by deployment. The open-source MLflow custom-scorer documentation says code-based scorers are unsupported by its automatic evaluation feature. Check the deployment you are using before assuming a Databricks-managed monitoring example will work unchanged.

Give each metric a precise meaning

Before adding a scorer, write down its pass condition and the evidence it needs. A required field, known expected argument, or explicit policy limit usually gives you a direct check. Assessing whether a free-form answer addresses an ambiguous request may justify a judge.

In the return-policy example, has_citation=True means the citation field meets the contract. Whether the policy supports a 90-day return window remains a separate question, with a separate assessment.

Kedro Meets the Lakehouse: Rebuilding an Real World Evidence Pipeline on Databricks

Salman_Ahmed — Mon, 07 Sep 2026 19:40:58 GMT

A few months back my team at Genpact picked up a real world evidence project for a life sciences client that had been running on Kedro for a couple of years. The pipeline touched patient level claims and EHR extracts, ran feature engineering for a handful of outcomes models, and trained models that fed into a reporting layer. It worked, but it had grown the way these things always do. New analysts kept adding nodes, config files multiplied across environments, and nobody outside the original team wanted to touch the DAG. When management asked us to consolidate everything onto Databricks, the first question wasn't whether Kedro was good or bad.

"It was how much of it we actually needed to keep."

Then came the final nail in the coffin, the Kedro Project was flagged at Client Governance monthly meet as a project which is using older versions of python libraries, and with many packages which are not supported by the latest version of python on serverless cluster setup. While the issue could be mitigated by packaging and wrapping the code in a Databricks Assets Bundle which needed another review round for compliance checks and permissions.

Why we didn't just rip it out

Kedro gave this project something worth preserving: pipelines that are testable, versioned, and easy to reason about node by node. Ripping that out and rewriting everything as raw PySpark scripts would have meant redoing months of validation work that regulatory reviewers had already signed off on. So the plan wasn't a rewrite. It was a migration where Kedro's pipeline structure stays as the mental model, but the runtime, storage, governance, and scheduling all move onto Databricks native services.

Kedro's own docs describe three ways to run a project on Databricks, and we ended up borrowing pieces from all three rather than picking just one, which the docs actually acknowledge is normal depending on where your team is in the project lifecycle.

The four pieces that mattered

GitHub as the source of truth. The Kedro repo stayed on GitHub, and we connected it into Databricks using Git folders instead of copying code into the workspace by hand. This kept every change to a node or a catalog entry reviewable through a pull request, and it meant the workspace was never the place where code actually lived. Databricks has a whole pattern for this called CI/CD with Git folders, including a production folder that only updates when a PR merges, which is exactly the discipline you want on a project touching PHI.

Serverless compute for the jobs themselves. We stopped provisioning and babysitting job clusters. Serverless compute for workflows lets Databricks handle scaling and node selection per task, and for a pipeline with uneven load between the ingestion nodes and the training nodes this removed a lot of guesswork around cluster sizing that used to eat up sprint time. It's also just less for a life sciences client's infra team to sign off on from a security review standpoint, since there's no persistent cluster configuration to audit. In addition by migrating we could remove legacy code to maintain hooks which keep an spark session active.

Unity Catalog for governance, not just storage. This was the part that actually mattered most given the PHI involved. Every Kedro dataset in the catalog.yml got remapped from local paths to Unity Catalog tables and volumes, which meant column level tagging, masking policies on direct identifiers, and row level filters became possible in a way that flat files in a Kedro data folder never allowed. Databricks has published a fair amount on this specifically for healthcare, and a lot of it lines up with basic HIPAA minimum necessary principles once you translate it into catalog language.

MLflow for the model side. The training pipeline still runs as a set of Kedro nodes, but every run now logs to MLflow, and the registered models live in the Unity Catalog model registry instead of a pickle file sitting in a shared drive. MLflow made this a bit smoother because metrics and parameters across experiments show up on one page instead of forcing you to hunt through separate runs, which matters when a reviewer wants to see how a model's performance changed across retraining cycles.

Databricks Jobs for orchestration. The Kedro pipeline gets triggered as tasks inside a Databricks Job rather than a cron entry calling kedro run on a VM somewhere. Lakeflow Jobs handles the scheduling, retries, and alerting, and because it's tied to the same workspace as Unity Catalog and MLflow, the lineage from raw ingestion through to a registered model is visible in one place instead of stitched together from logs.

What we didn't solve on day one

I want to be honest that this wasn't a clean, one sprint migration. Click based exit codes from the packaged Kedro entrypoint confused job status reporting for a few weeks until we adjusted how failures were surfaced. Some non-Spark datasets needed path rewrites we didn't catch until a run failed on a missing local folder that used to exist in the old VM setup. And cloning this project structure for a second requirement is still a manual exercise, we're building an internal template for that now rather than trusting copy paste inside the workspace to preserve config isolation between clients.

The part worth remembering

None of this was about choosing Databricks over Kedro or the other way around. It was about letting each tool do the part it's actually good at. Kedro still owns the pipeline logic and the testing discipline that got this project through validation in the first place. Databricks owns the parts that used to be the actual pain, which cluster to use, who can see which column, and where a model's lineage actually lives. For a project handling patient data, that second half turned out to matter a lot more than we expected going in.

References

Kedro documentation on Databricks deployment patterns: https://docs.kedro.org/en/stable/deploy/supported-platforms/databricks/
Databricks community thread on running Kedro projects: https://community.databricks.com/t5/forums/filteredbylabelpage/board-id/data-engineering/label-name/pipelines
CI/CD with Databricks Git folders: https://docs.databricks.com/aws/en/repos/ci-cd
Run Lakeflow Jobs with serverless compute: https://docs.databricks.com/aws/en/jobs/run-serverless-jobs
Automating governance of PHI data in healthcare (Databricks blog): https://www.databricks.com/blog/automating-governance-phi-data-healthcare
Model Registry improvements with MLflow 3: https://docs.databricks.com/aws/en/mlflow/model-registry-3
Lakeflow Jobs overview: https://docs.databricks.com/aws/en/jobs/

🚀 My Lakebase App is Working!

Raunak567 — Mon, 07 Sep 2026 18:48:23 GMT

I successfully deployed my app and fixed the database permission issue.

The main issue was with permissions in Overview → Roles & Database. I found a separate randomly generated role/number associated with the database, and after granting it the required access, my app was able to query the todos table successfully.

For troubleshooting, I used Google Gemini as my AI agent. I provided the error message and my code, and used its suggestions to understand what was causing the database access problem.

Agent used: Google Gemini
Prompt used:
“I'm getting a failed to get todos error in my deployed Databricks Lakebase app. Here is my code and the error message. Help me identify the database permission issue and explain how to fix it.”

Now the app is working successfully!