Databricks Community

Why your materialized view keeps doing full recomputes, and how to find out

This came out of a community thread about serving gold materialized views to Power BI. Everything downstream kept getting rewritten, and the root cause turned out to be simple: most of the MVs were being fully recomputed on every update. A full recom...

Federated PostgreSQL timestamps in Unity Catalog: a view that gives everyone the same answer

This came out of a community thread about a foreign catalog over PostgreSQL where daily counts in Databricks didn't match Postgres, and setting the session time zone made things more confusing, not less. I got part of it wrong the first time, so here...

Introducing 5XL SQL Warehouses: A Practical Guide to Meeting SLAs for Your Most Demanding Workloads

While data volumes are growing, the time windows to process them are getting stricter. Enterprise customers are generating more data than ever, and the speed is only accelerating with more agentic workflows. At the same time, the SLAs for when it nee...

Transactional apps on Databricks: Lakebase owns the decision, Delta gets the copy

A condensed version of the full article Beyond Analytics: Building Transactional AI Applications with Databricks and Lakebase. The working demo and its tests are in lakebase-patch-desk.The pipeline is built and the numbers are in Delta. Now business ...

Genie agents and Unity Catalog wikis: what I tested, and what to use instead

I tested whether a Genie agent picks up a business rule from a Unity Catalog wiki (Beta). Short answer: not today. A Genie agent works from the data, the table and column comments, and the context you configure in the agent. Here's the test and what ...

Announcement | The new Genie: autonomous AI for complex enterprise work

Genie is evolving into an autonomous AI coworker that can take on long-running enterprise tasks, not just answer one-off questions. It can monitor progress, follow up when information is missing, and act as new information arrives, within the guardra...

Private Connectivity from Databricks Serverless on AWS to Your On-Prem Data Center

How to connect Databricks serverless workloads to on-premises databases and services over a private network path, using Network Connectivity Configurations and AWS PrivateLink. Why private connectivity to on-prem matters With serverless compute, you ...

CUSTOMER STORY | Mural answers data questions in minutes with Genie Agents

“With Genie, people are asking a lot more questions. They're poking around, experimenting and finding insights no one thought to look for before.” - Wendy Lam, Director of Data Engineering, Mural Mural gives companies like IBM, SAP and Deloitte a sha...

From Kyriba to Lakehouse: A Near-Real-Time Treasury Data Pipeline on Databricks

The two-speed problem of treasury data Corporate treasury asks the same question at two speeds. Where is our money right now? And where was it at the close of business? For many treasury teams the answer lives in Kyriba. Cash flows, balances and FX e...

How to Query Unity Catalog Views from External Engines

Summary External engines can now query Unity Catalog views. Views over Delta tables are served through the Unity REST API (Generally Available). Views over Iceberg tables are served through the Iceberg REST Catalog (Beta).The external engine fetches ...

Announcement | Genie Ontology is Now Generally Available

Genie Ontology is now Generally Available, a shared business context layer built from the work teams already do, governed by people, and designed to help Genie and connected agents understand how your organization works. Key highlights Learns from ev...

Unity Catalog secrets: what each grant does, and the schema-level READ SECRET that fails

I tested Unity Catalog secrets in my own Azure Databricks sandbox (serverless, environment version 6) to see what each privilege actually lets a service principal do. Most of it matches the docs. One thing doesn't: READ SECRET can't be granted on a s...

Reinforcement Learning on Databricks: Building a Contextual Bandit

Is MLflow really built only for supervised learning, or can it handle something like reinforcement learning? A customer asked me exactly this. They're running a contextual-bandit-style system on-premises and are thinking about moving it to the cloud ...

Getting started with DQX

Announcement | Genie with Genie Ontology delivers 1.6× the accuracy with less than half the tokens

A new enterprise benchmark developed with Turing compares Genie with two frontier agents on business questions in a live Databricks workspace. On this benchmark, Genie achieved a 91% pass rate, compared with 45% and 58%, while using less than half th...