All Genie Hub posts

Re: Learn Databricks Genie: 5 courses to go from curious to certified

Tushar_Parekar — Thu, 13 Aug 2026 09:38:03 GMT

Thanks for the suggestion, @Nidhig631! We're working on a similar roundup for Agent Bricks, stay tuned!

Re: Learn Databricks Genie: 5 courses to go from curious to certified

Nidhig631 — Thu, 13 Aug 2026 02:39:01 GMT

This is really good, Thanks for sharing.

Can you share similar listing for agent bricks.

Learn Databricks Genie: 5 courses to go from curious to certified

Tushar_Parekar — Wed, 12 Aug 2026 17:33:56 GMT

Databricks Training & Certifications

Learn Databricks Genie

Want to turn plain-language questions into governed, secure answers from your data? Databricks AI/BI Genie is the conversational analytics layer that lets business users ask questions and get trustworthy, SQL-backed results - no code required.

The Databricks catalog currently lists 5 Genie courses - a mix of free and paid, from a 2-hour onboarding to instructor-led Associate labs. Pick the one that fits your level and start learning. 👇

Start here

FREE

Answering Business Questions with Databricks Genie

Learn what a Genie Space is, how it keeps your data secure, and best practices for great results. Closes with a hands-on game where you compete to investigate a business mystery.

Self-paced 2 hours Onboarding

Enroll free →

FREE

Business Impact Accelerator: Answering Business Questions with Genie

A guided, instructor-led take on Genie for business users - ask plain-language questions, understand governance and security, and learn best practices in a live lab setting.

Instructor-led + Lab 3 hours Onboarding

Enroll free →

Go deeper: build production Genie Agents

Building Reliable Conversational Agents with Genie teaches the full end-to-end workflow - defining benchmarks, curating the Knowledge Store (metadata, synonyms, SQL logic, example queries), sharing via Genie One, and enforcing Unity Catalog governance. Available in three formats:

Self-paced · Onboarding

FREE 2 hours

Start free →

Self-paced + Lab · Associate

PAID / SUBSCRIPTION 3 hours

View course →

Instructor-led + Lab · Associate

PAID 4 hours

View course →

Browse all Genie courses in the catalog →

Governed. Secure. Conversational. Learn Genie the right way with official Databricks training.

#Databricks #Genie #AIBI

Integrate genie workspaces automatically based on connection.

Aadarsh_K — Wed, 12 Aug 2026 13:48:12 GMT

Hi Genie Hub team,

I’ve been building an enterprise application around Genie and ran into an interesting integration problem that I’d love to get the community’s perspective on.

In our setup, the application can connect to different Databricks environments, but integrating Genie currently means explicitly managing the Genie Space/workspace configuration for each environment, while also dealing with Unity Catalog permissions, access control and governance.

That becomes increasingly difficult when the same application needs to support multiple customers or multiple Databricks workspaces.

This led me to a broader idea:

Could Genie integration become connection-driven rather than configuration-driven?

For example:

Databricks connection → discover eligible Genie Spaces → validate UC/access requirements → automatically select/register the appropriate Genie Space → enforce the required governance model.

The interesting part for me is that the application shouldn't need to know “use Genie Space ID X for workspace Y.” The Databricks connection itself should be enough to discover and establish the correct Genie integration.

I’m currently experimenting with this architecture and would particularly like to hear from people working on Genie at scale:

How are you handling Genie Space discovery across multiple workspaces?
How are you managing UC permissions and governance for application-driven Genie access?
Is there a recommended Databricks-native pattern for making this connection/configuration dynamic?
Would an automated connection → Genie integration layer be useful for enterprise deployments?

I’d be very interested in connecting with anyone from the Genie/AI-BI or Databricks partner ecosystem who is working on this problem. I’d also be happy to share what we’ve built and the problems we encountered in a real implementation.

Re: Dashboard for Genie Cost and Usage tracking

Shahid_Haider — Tue, 11 Aug 2026 12:04:08 GMT

Very useful—really appreciate your work. Thank you!

Solution: Simplify Genie Agent Instruction Updates Across Multiple Spaces

harisrinivasay — Tue, 11 Aug 2026 05:54:18 GMT

Dear All,

As the number of Genie Agents grows, maintaining consistent instructions across multiple Genie Spaces becomes increasingly cumbersome. Updating the same instructions manually in each space requires significant effort and is both time-consuming and error-prone.

To simplify this process, we have created a notebook-based solution that automatically updates Genie instructions across multiple spaces. To use it, simply replace the following values in the notebook:

Genie Space IDs
Databricks Workspace URL
Personal Access Token (PAT)
Desired Instructions

Once configured, the notebook will automatically apply the same instructions to all specified Genie Spaces, eliminating the need for repetitive manual updates.

This should significantly reduce maintenance effort and ensure consistency across your Genie Agents.

Attached notebook code:

================================start here====================================

import requests
import json

workspace_url = "<your databricks workspace url"
token = "<your PAT token"

headers = {
"Authorization": f"Bearer {token}",
"Content-Type": "application/json"
}

space_ids = [
"<genie space id 1>",
"<genie space id 2>",
"<genie space id n>
]

NEW_INSTRUCTION = """
Your Role: Databricks SQL Analyst.
Rule 1: Use only tables, columns, joins, and logic defined in this Genie space. Never invent any.
Rule 2: Before generating a query, validate all rules.
<Your extended prompt Rules>
"""

for space_id in space_ids:
try:
# Get existing space
url = f"{workspace_url}/api/2.0/genie/spaces/{space_id}"
response = requests.get(
f"{url}?include_serialized_space=true",
headers=headers
)

response.raise_for_status()

space = response.json()

serialized = json.loads(space["serialized_space"])

# Replace instructions
serialized["instructions"]["text_instructions"] = [{
"content": [NEW_INSTRUCTION]
}]

payload = {
"serialized_space": json.dumps(serialized)
}

update_resp = requests.patch(
url,
headers=headers,
json=payload
)

print(
f"{space_id}: {update_resp.status_code}"
)
except requests.exceptions.RequestException as e:
print(f"{space_id}: Request failed - {e}")
except Exception as e:
print(f"{space_id}: Unexpected error - {e}")

=========================================end here================================

Copilot for Data Platform Operations - Genie ZeroOps (24 * 7)

rkhand14_ltm — Mon, 03 Aug 2026 12:06:17 GMT

Databricks Genie ZeroOps is one of the most interesting announcements from Data + AI Summit 2026. It is essentially an AI-powered autonomous DataOps agent built directly into the Databricks platform that continuously monitors your data and AI workloads, diagnoses issues, and proposes fixes automatically.

Genie ZeroOps is designed to work in the background 24x7, significantly reducing the operational burden on data engineering teams. Rather than simply alerting teams when issues occur, it proactively:

Monitors data and AI workloads continuously
Detects failures and anomalies
Diagnoses the root cause using lineage and observability data
Generates remediation recommendations
Validates fixes in a secure sandbox environment
Routes changes for human approval before production deployment

High level approach from Reactive Operations to Autonomous Operations.

Detect

↓

Auto RCA

↓

Auto Fix Generation

↓

Sandbox Testing

↓

Human Approval

Exciting to get Genie in GA.

Re: How to get started with Genie Ontlogy?

szymon_dybczak — Wed, 29 Jul 2026 10:04:52 GMT

Hi ,

What are the prerequisites for using Genie Ontology?

Genie Ontology is under gated Public Preview. If you want to try it you need to contact with Databricks account team

How do I build and configure an ontology from scratch?

Rather than configuring the ontology object directly, you invest in the Unity Catalog semantic layer that feeds it: Glossary for authoritative concepts and terms, Domains to organize data and AI assets into business-aligned categories, and Metrics to govern KPIs once as reusable objects.

Are there any official guides, best practices, or sample implementations that you would recommend?

Not yet. This is really new topic and access is only for selected ones. You can check some blogs:

Introducing Genie One, Genie Agents, and Genie Ontology | Databricks Blog

Here is some video example: Databricks DAIS 2026 ALL ANNOUNCEMENTS SUMMARIZED | Ontology, Reyden, LTAP, Omnigent & More

Is Genie Ontology currently in Public Preview, Private Preview, or General Availability?

Genie Ontology is under gated Public Preview. If you want to try it you need to contact with Databricks account team

Where should I be able to access it in the Databricks workspace? I don't see the Genie Ontology option or icon in my workspace, so I'm wondering if there are any workspace requirements, feature flags, or permissions that need to be enabled.

It should be visible under Genie One. Check above youtube video. You don't see it because public preview is gated and you need to contact databricks team to enable it.

If my asnwer was helpful, please consider marking it as accepted solution

Dashboard for Genie Cost and Usage tracking

DataCloudy — Wed, 29 Jul 2026 07:33:41 GMT

Sharing my new dashboard for tracking Genie usage and costs, suggestions for improvements and features are welcome.

The current version displays and aggregates Genie LLM costs, free & billable DBU consumptions, and code interactions per user on a daily, monthly, and workspace leve. More functionality will be added soon, it can be found at https://github.com/WowdyCloudy/wowdycloudy/tree/main/dashboards

How to get started with Genie Ontlogy?

Niyojit — Wed, 29 Jul 2026 07:04:40 GMT

Hello Databricks Community,

I recently started exploring Genie Ontology and would like to implement it as part of a POC. However, I'm not sure how to get started.

I have a few questions:

What are the prerequisites for using Genie Ontology?
How do I build and configure an ontology from scratch?
Are there any official guides, best practices, or sample implementations that you would recommend?
Is Genie Ontology currently in Public Preview, Private Preview, or General Availability?
Where should I be able to access it in the Databricks workspace? I don't see the Genie Ontology option or icon in my workspace, so I'm wondering if there are any workspace requirements, feature flags, or permissions that need to be enabled.

If anyone has already worked with Genie Ontology, I'd really appreciate any guidance, documentation, or tips on getting started.

Thank you!

Re: UC Permission Automation Strategy

samson01 — Fri, 24 Jul 2026 02:21:27 GMT

Thank you @binlogreader your perspective and experience were very insightful

Re: UC Permission Automation Strategy

binlogreader — Fri, 24 Jul 2026 01:58:44 GMT

We run Terraform as the system of record for Unity Catalog grants, with asset bundles handling a separate layer, and the split of responsibilities turned out to matter more than the tool choice.

The first lesson cost us real debugging time. databricks_grants is authoritative for the securable it points at. It manages the complete grant list on that object, so if two stacks or repos both declare grants on the same catalog, every apply of one silently reverts the other. The symptom is grants that seem to disappear randomly. The fix was organizational rather than technical, exactly one stack owns each securable's grants, and every other team adds grants by pull request to that owner.

Second, humans get access only through groups, with membership managed in the identity provider and synced down. Day-to-day access changes are then membership operations that never touch Terraform. The code changes only when a new group-to-privilege relationship appears, which is rare and worth the review. Service principals get their own explicit grants, since those represent workloads, not people.

Third, your Terraform and metadata-driven options are less opposed than they look. Grant definitions in Terraform are already metadata, maps of principal to privileges fed into for_each, and you can keep that config in YAML files the code reads. That gives you the centralized declarative model of the API approach while keeping plan diffs, code review, and drift detection for free, all things a custom REST engine has to rebuild. In our experience the custom engine only earns its complexity when you need approval workflows or self-service at a scale where pull requests become the bottleneck.

One trap to know about. Account-level roles are not databricks_grants at all but databricks_access_control_rule_set, and every apply replaces the whole rule set it names. Keep all grants for a given rule set path in one resource, and import anything that predates Terraform, because an apply that doesn't include an existing grant removes it.

Manual UI grants we treat as break-glass only. With one owner per securable, anything added by hand shows up in the next plan and either gets codified or reverted, which is exactly the audit trail you want.

How many teams will be requesting access in your setup? The single-owner pull request model holds up well to a point, and past that point the metadata layer in front of Terraform is where I'd invest.

UC Permission Automation Strategy

samson01 — Fri, 24 Jul 2026 00:01:08 GMT

Hello Community,

I’m starting this discussion to gather perspectives and best practices for managing Unity Catalog (UC) permissions across all securables, particularly from a platform governance, automation, and operational scalability standpoint.

I have encountered differing opinions on the most effective approach for governing UC permissions:

Terraform – Provides infrastructure-as-code consistency, but can introduce significant complexity when managing nested permissions, inheritance models, and isolation boundaries at scale.
REST APIs with a metadata-driven approach – Managing permissions through YAML, JSON, or a Lakehouse-backed configuration model, enabling centralized governance and stateful automation.
Manual administration through the UI – Simpler for smaller environments, but potentially challenging to scale, audit, and maintain consistency across platforms.

I would be interested in learning:

How is your organization governing Unity Catalog permissions today?
What automation patterns have proven successful?
How are you balancing governance, operational overhead, and flexibility?
What lessons learned or pitfalls would you recommend avoiding?

I appreciate any insights, experiences, or recommendations the community can share.

Thank you for your contributions.

How good is the new /ImportBI Genie Skill?

KrisJohannesen — Tue, 07 Jul 2026 09:31:30 GMT

After some initial testing on small models which were quite impressive, I decided to put the new Genie skill to the full test with a more advanced model in Power BI.

Be aware that the /ImportBI feature is in Beta. This means that there are ongoing changes happening, and that the final product might be different to what you are experiencing today!

Below, you can catch the full 45 minute experience of me playing around with in on YouTube. You get all of my honest takes, my confusions when something does not work out the way I think, as well as a lot of me rambling about on some of the things I find along the way.

However, if you want the slightly shorter recap, this article might suit you better!

Setting up the model

First, I created a semantic model and report myself using the Tabular Editor SpaceParts dataset, that you can find on the Databricks Marketplace. Huge shoutout to TE team for putting a real-scale demo dataset out there, that has a great coverage of data, as well as some intentional design "mistakes" that you might find in the real world!

Then, after showing it to my colleague David Kofod Hanna, he was not too impressed with my out-of-data Power BI skills. Lucky for me (and you!) he was kind enough to help me develop a better model and report to run the tests on. Massive shoutout to David for that!

A quick overview of the report pages can be seen in the below animation. The model itself contains 4 facts and 11 dimensions, as well as some more advanced things that are highlighted below. For visualizations we use the following:

KPI Cards (with categories, and reference values/labels)
Slicers (Regular and Input)
Matrix Table (using Field Parameters - more on those below)
Bar Chart (Clustered)
Area Chart
... and then some buttons/text boxes if you count those as well.

Some advanced areas to test

So, in order to devise a proper test, the Power BI model contains a couple of elements that are essential to the way we develop and work with Power BI on a larger scale, I would consider them best practices at least in our ways of working at twoday. Some of these are more visual-based, while others are more focused on the model itself. The ones I want to highlight in this test case are:

Calculation Groups; In this case there are two. One for Time Intelligence, and one for Unit Conversion that are used across different visuals
Field Parameters; these are used for running the Pivot Table Analyze page, in which users can switch between measures and dimensions themselves in the Pivot Table
Model Documentation; the model has a built-in documentation page using the INFO.VIEW DAX functions. This tells the end-user how things are calculated as a sort of business glossary.
Information measures; here I am referring to the concept of adding measures to show e.g. Last Refresh, Filters Applied and other nice-to-know information

Understanding how /ImportBI works

The primary use case for ImportBI is teams already running Power BI or Tableau with Databricks as their primary data source. If your M queries simply point to Databricks catalog tables with no transformation logic, you're in the best position to benefit.

ImportBI is triggered via the Skill /importBI inside of Genie Code. It works mostly autonomously and assists in analyzing the model and planning what needs to be done. After approval it then translates measures, and builds dashboards in parallel before presenting you with the final output in both code and a small report.

If you have not seen the long-form video, and want a quicker rundown of how ImportBI works, this 5 Minute Features version might be more your style. In the video, I break down what ImportBI does and how, using a very simple Power BI model.

The workloads carried out by ImportBI can be split into 3 phases, as described below:

Setup is the first phase, in which we instruct Genie in what type of BI solution we have (PowerBI or Tableau) as well as direct it to a volume where it can find the file. In case of PowerBI it relies on the PBIT format. We also supply a catalog and schema for both the source (where the data lives) and the target (where it will create the Metric Views).
Analyzing and Planning is the phase where Genie unpacks and understands the existing model. It breaks down both the model itself, but also the visual side, and writes up a short report breaking down what it plans to do with each part - including anything that it will omit from the migration - either because it is unused, or because it is not directly migratable.
Metric Views and Dashboards is the actual migration phase. After approval of the plan in the previous step, Genie gets to work. It starts by developing the Metric Views, and then subsequently it does the Dashboard work.

As with any Genie Code session, you can intervene, redirect or change the plan and work carried out along the way. The above is just the general path that the skills takes with no redirection or additional instructions.

The results are in

So, as you know by now, I have run two different tests. One on a simple model, and one on a complex model. This will mainly discuss the results of the complex one, as that is by far the most interesting.

We will cover the Semantic Layer and the Visual Layer separately, and then sum up, evaluate, and then you can find my final verdict at the bottom.

Semantic Model to Metric Views

What works well, is understand the wiring between Databricks and Unity Catalog and does the basics:

The M expressions are parsed, and broken down to their correct counterpart catalog.schema.table setup, and added to the Metric Views sources and joins. I am unsure how much of this is derived from Unity Catalog directly, or if e.g. join columns are derived from the model design in PowerBI
Each Fact Table is created as its own metric view, all the fields are correctly identified
All generic/simple measures are created correctly
It references cross-measure references, and creates measures using other measures
Includes the original DAX expression as a comment to the new measures - ensuring traceability
The active filters footer and last refresh timestamp measures are not re-created. This might seem like a drawback, but actually both are built into the AIBI Dashboard UI natively which is why they are not there.

What does not work so well, it is honestly almost all of the more advanced stuff, as well as a few basics:

There is no Calculation Group equivalent concept in Databricks Metric Views. The workaround, that I myself prompted Genie to do, is creating static YTD, QTD, and MTD measures per metric. However, this contradicts the dynamic "inject any measure into any time window" flexibility of Calculation Groups.
Field Parameters are completely omitted - there is no style of dynamic parameter-based
Cross-fact measures e.g. comparing values across fact tables isn't currently possible. Dashboard relationships are in private preview, and might enable this soon though.
USERPRINCIPALNAME() as a measure is not handled. As far as I know, a workaround for a similar meaure exists in Databricks but the tool doesn't auto-detect it.
Only one of the four created Metric Views was added to the dashboard, the others had to be added manually. I believe this is because it created some of the measures on the wrong facts - therefore not needing the other ones directly.

Power BI Dashboard to AIBI Dashboard

What works well, is understand the visuals chosen and selecting the right fields:

The classic BI visuals are replaced correctly, choosing card, bar chart and area chart, as well as the slicers. This works for almost all of the visuals, only one is wrong.
The right measures/dimensions are added to the visuals

What does not work so well, is the actual attention to detail:

The visual placement is all over the place. The filters are below the visuals and the grid-style of AIBI Dashboards limits the flexibility a bit.
Genie added cards that did not exist in the original layout. Not entirely sure why, as there is not indication we would want invoice Cost and Quantity.
The filters are not all added on both pages. Despite Global filters being a thing, the Genie Code approach was to add them directly on page. Not a problem in itself, but it split them, adding two on each instead of all four filters on both
Power BI's ability to split a single card visual by a dimension into multiple cards doesn't exist in AI/BI dashboards. For some reason, it decided to do a Bar-chart instead.
The two later pages with Documentation and Pivot Table are not created at all. This is not a visual problem, as they are not created because there is not a direct replacement for them on the semantic model level. I did however manage to do the Documentation - catch that at the end!

The re-created AIBI Dashboard pages. As you can tell, they are not really up to scratch.

My personal verdict

For simple models with built directly on Databricks, with no advanced DAX patterns, ImportBI can meaningfully accelerate the transition. However, how many of those type of solutions do you really have in an enterprise context? For anything production-grade using features like time intelligence, field parameters, complex visual layouts, cross-fact calculations, you should expect a significant amount of manual work after the import.

For now, ImportBI is a useful starting point, not a full-scale migration tool

Remembering that this is still a Beta feature, I know my critique might seem a little harsh. I sincerely hope that some of these kinks get worked out, and honestly, some of them might even already be fixed when you read this.

To me, some of the most jarring things that need to get worked out is actually in the Dashboard part. Filters that are left out on one page but added to another. Visuals that are not placed correctly, or are not the right type is a real issue. The fact that it does not seems to understand when to add what to a page is a real issue when it claims to be able to do something.

On the Metric Views side, there are some obvious shortcomings related to cross-fact measures that needs to be solved elsewhere, but other than that, most of this part seems solid. I would however love to see some sort of a dynamic approach to time intelligence and/or parameter swapping as part of the intersection between the dashboard and metric views experience.

The important part, to me, is that tool is self-aware of its gaps and shortcomings. During the planning phase, it flags what it's skipped and why, and its migration report gives you a clear checklist of what needs your attention. That level of transparency is something I can appreciate - and it also serves as a good guide for what Databricks might choose to work on in the future.

My overall verdict of this as a migration tool for Power BI workloads is a score of: 4/10

... an unexpected bonus: Model Documentation

One useful discovery I made, was that we are actually able to do the documentation setup with only a tiny bit of additional work.

This was done in parallel, in another Genie session, but proved to be quite valuable nonetheless. The main point here being that ImportBI had no actual stake in this.

Because Databricks stores Metric View definitions in a queryable format, we are able to easily extract the YAML definition in code. This is done by querying the information_schema tables. From here we can extract each measure's expression, description, comment and window function configuration, directly from the YAML.

Then, by writing this to a Delta table, slapping a Metric View on top, and adding a simple Table Visual in a Dashboard, you can get a dynamic documentation layer, as shown below:

I believe the Documentation topic deserves a bit of additional attention. So expect a full write-up on it later on with all the code included for you to work with it yourself!

Re: Building Production-Ready SDP Pipelines with Genie Code: The Complete Guide

Mycimmunity — Tue, 23 Jun 2026 00:23:22 GMT

Great Overview, Thanks for sharing with the community.

Re: Databricks now supports importing Tableau and Power BI files into Genie Code to automatically bu

anshul-mehta — Mon, 22 Jun 2026 08:06:46 GMT

How does databricks handle calculated fields etc In TWB files they are referenced as calculation_1234.... does databricks store these references at the data model level to resolve them when I upload a TWB file

Building Production-Ready SDP Pipelines with Genie Code: The Complete Guide

shwetav1407 — Tue, 16 Jun 2026 18:22:36 GMT

How Databricks’ AI agent transforms data engineering from manual craftsmanship into conversational pipeline development

Data engineers have long accepted a painful truth: building production-grade ETL pipelines means wrestling with hundreds of lines of orchestration code, manually encoding execution order, handling incremental processing logic, and then praying nothing breaks at 2 AM. Spark Declarative Pipelines (SDP) already simplified this dramatically by letting you declare what your data should look like rather than how to get there. Now, with Genie Code in Agent mode, you don’t even have to write those declarations yourself.

Press enter or click to view image in full size

In this guide, we’ll walk through building a complete medallion architecture pipeline using Genie Code and SDP — from raw ingestion through business-ready analytics — and explore the patterns that make this approach production-worthy.

What Is SDP, and Why Should You Care?

Lakeflow Spark Declarative Pipelines (SDP) is Databricks’ framework for building batch and streaming data pipelines in SQL and Python. Unlike traditional Spark jobs where you manually define execution order, manage checkpoints, and handle retries, SDP lets you declare your transformations and handles the orchestration automatically.

The key benefits that matter for real-world pipelines:

Automatic orchestration — SDP analyzes dependencies across all your source files, builds a dataflow graph, and determines the optimal execution order with maximum parallelism. It also retries failures at the most granular level possible: first the Spark task, then the flow, then the pipeline.
Incremental processing built in — Materialized views automatically process only new data and changes. No more writing MERGE statements by hand.
Data quality as code — Expectations let you define quality constraints inline, right next to your transformations.
Unified batch and streaming — Toggle between batch and streaming processing modes with a single keyword change.

Here’s what that looks like compared to traditional approaches:

The Old Way (PySpark + Manual Orchestration)

# Hundreds of lines for a simple weekly sales pipeline
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, sum, window
from delta.tables import DeltaTable

spark = SparkSession.builder.getOrCreate()

# Step 1: Read raw data (manually handle incremental)
raw_df = spark.read.format("delta").load("/data/raw_sales")
last_processed = spark.read.format("delta") \
    .load("/checkpoints/last_ts").collect()[0][0]
new_data = raw_df.filter(col("event_time") > last_processed)

# Step 2: Clean (manually write quality checks)
cleaned = new_data.filter(
    col("amount").isNotNull() & 
    (col("amount") > 0)
)

# Step 3: Aggregate (manually handle upserts)
weekly = cleaned.groupBy(
    window("event_time", "1 week"), "region"
).agg(sum("amount").alias("total_sales"))

# Step 4: Write (manually handle merge)
target = DeltaTable.forPath(spark, "/data/weekly_sales")
target.alias("t").merge(
    weekly.alias("s"),
    "t.window = s.window AND t.region = s.region"
).whenMatchedUpdateAll().whenNotMatchedInsertAll().execute()

# Step 5: Update checkpoint (manually track state)
# ... plus an Airflow DAG for scheduling, retries, alerting

The SDP Way (SQL)

-- The entire pipeline in a few declarations

-- Bronze: raw ingestion with Auto Loader
CREATE OR REFRESH STREAMING TABLE bronze_sales
AS SELECT * FROM STREAM read_files(
  '/data/landing/sales/',
  format => 'json',
  schema => 'event_time TIMESTAMP, region STRING, 
             product STRING, amount DOUBLE'
);

-- Silver: cleansed with quality expectations
CREATE OR REFRESH STREAMING TABLE silver_sales (
  CONSTRAINT valid_amount EXPECT (amount > 0) ON VIOLATION DROP ROW,
  CONSTRAINT not_null_region EXPECT (region IS NOT NULL) ON VIOLATION DROP ROW
)
AS SELECT
  event_time,
  region,
  product,
  amount,
  current_timestamp() AS processed_at
FROM STREAM(bronze_sales);

-- Gold: business-ready weekly aggregation
CREATE OR REFRESH MATERIALIZED VIEW gold_weekly_sales
AS SELECT
  date_trunc('week', event_time) AS week_start,
  region,
  COUNT(*) AS transaction_count,
  SUM(amount) AS total_sales,
  AVG(amount) AS avg_transaction
FROM silver_sales
GROUP BY date_trunc('week', event_time), region;

That’s it. SDP handles incremental processing, execution order, retries, and checkpoint management. The bronze and silver tables use streaming semantics (the STREAM keyword), while the gold materialized view uses batch semantics but still only reprocesses changed data.

Enter Genie Code: Your AI Data Engineering Partner

Now here’s where it gets interesting. Genie Code in Agent mode — available inside the Lakeflow Pipelines Editor — doesn’t just help you write SDP code. It can autonomously plan, generate, run, validate, and fix entire pipelines from a single natural language prompt.

How Genie Code Agent Mode Works

When you enable Agent mode in the Genie Code panel within the Lakeflow Pipelines Editor, the agent adapts its capabilities specifically for data engineering tasks. Unlike chat mode, Agent mode can:

Plan a multi-step solution and present it for your review
Search your Unity Catalog for relevant tables, schemas, and lineage
Generate SQL or Python SDP source files in the pipeline editor
Run pipeline updates and read the output datasets
Diagnose and fix errors automatically, iterating until the pipeline succeeds
Respect your Unity Catalog permissions — it can only access data you can access

The key design principle is human-in-the-loop: Genie Code proposes plans and asks for approval before executing. You can Allow, Decline, or ask it to try a different approach.

Tutorial: Building a Medallion Pipeline with Genie Code

Let’s walk through building a real pipeline — an e-commerce analytics pipeline that ingests order data, cleans and enriches it, and produces dashboards-ready metrics.

Prerequisites

A Databricks workspace with Partner-powered AI features enabled
Access to the Lakeflow Pipelines Editor
Unity Catalog configured with a target catalog and schema

Step 1: Create Your Pipeline and Open Genie Code

Navigate to Pipelines in the sidebar and create a new pipeline. Give it a name like ecommerce_analytics and set your target catalog and schema (e.g., analytics.ecommerce).

Once in the Lakeflow Pipelines Editor, open the Genie Code panel and switch to Agent mode.

Step 2: Prompt Genie Code to Build the Pipeline

Start with a descriptive prompt that tells Genie Code what you want:

Your prompt: “Build a medallion architecture pipeline for e-commerce analytics. I have raw order data landing as JSON files in /Volumes/raw_data/orders/ with fields: order_id, customer_id, product_id, quantity, unit_price, order_timestamp, and shipping_region. Create bronze ingestion with Auto Loader, silver cleansing with quality expectations, and gold aggregations for daily revenue by region and top products.”

Genie Code will create a step-by-step plan that looks something like:

Plan:
1. Search Unity Catalog for existing related tables
2. Create bronze_orders.sql — streaming table with Auto Loader
3. Create silver_orders.sql — cleaned data with expectations
4. Create gold_daily_revenue.sql — daily revenue by region
5. Create gold_top_products.sql — top products materialized view
6. Run the pipeline and validate outputs

Review the plan, ask clarifying questions if needed, then select Allow to let Genie Code proceed.

Step 3: Watch Genie Code Generate Your Pipeline

Genie Code creates each source file in your pipeline. Here’s what the generated code typically looks like:

File: bronze_orders.sql

-- Bronze layer: raw ingestion from JSON landing zone

CREATE OR REFRESH STREAMING TABLE bronze_orders
COMMENT 'Raw e-commerce orders ingested via Auto Loader'
AS SELECT
  *,
  _metadata.file_name AS source_file,
  _metadata.file_modification_time AS file_mod_time,
  current_timestamp() AS ingestion_timestamp
FROM STREAM read_files(
  '/Volumes/raw_data/orders/',
  format => 'json',
  inferColumnTypes => 'true'
);

File: silver_orders.sql

-- Silver layer: cleansed and validated orders

CREATE OR REFRESH STREAMING TABLE silver_orders (
  CONSTRAINT valid_order_id 
    EXPECT (order_id IS NOT NULL) ON VIOLATION DROP ROW,
  CONSTRAINT valid_quantity 
    EXPECT (quantity > 0 AND quantity < 10000) ON VIOLATION DROP ROW,
  CONSTRAINT valid_price 
    EXPECT (unit_price > 0) ON VIOLATION DROP ROW,
  CONSTRAINT valid_timestamp 
    EXPECT (order_timestamp IS NOT NULL) ON VIOLATION DROP ROW,
  CONSTRAINT valid_region 
    EXPECT (shipping_region IS NOT NULL) ON VIOLATION FAIL UPDATE
)
COMMENT 'Cleansed orders with quality expectations enforced'
AS SELECT
  order_id,
  customer_id,
  product_id,
  CAST(quantity AS INT) AS quantity,
  CAST(unit_price AS DOUBLE) AS unit_price,
  CAST(quantity AS INT) * CAST(unit_price AS DOUBLE) AS line_total,
  CAST(order_timestamp AS TIMESTAMP) AS order_timestamp,
  UPPER(TRIM(shipping_region)) AS shipping_region,
  ingestion_timestamp
FROM STREAM(bronze_orders);

File: gold_daily_revenue.sql

-- Gold layer: daily revenue metrics by region

CREATE OR REFRESH MATERIALIZED VIEW gold_daily_revenue
COMMENT 'Daily revenue aggregation by shipping region'
AS SELECT
  DATE(order_timestamp) AS order_date,
  shipping_region,
  COUNT(DISTINCT order_id) AS total_orders,
  COUNT(DISTINCT customer_id) AS unique_customers,
  SUM(line_total) AS total_revenue,
  AVG(line_total) AS avg_order_value,
  SUM(quantity) AS total_units_sold
FROM silver_orders
GROUP BY DATE(order_timestamp), shipping_region;

File: gold_top_products.sql

-- Gold layer: top products by revenue

CREATE OR REFRESH MATERIALIZED VIEW gold_top_products
COMMENT 'Product performance ranked by total revenue'
AS SELECT
  product_id,
  COUNT(DISTINCT order_id) AS times_ordered,
  SUM(quantity) AS total_units,
  SUM(line_total) AS total_revenue,
  AVG(unit_price) AS avg_price
FROM silver_orders
GROUP BY product_id;

Step 4: Genie Code Runs and Validates

After generating the files, Genie Code asks for permission to run the pipeline. Once you approve it:

Triggers a pipeline update
Monitors execution across all flows
Reads the output datasets to verify data landed correctly
Reports back with row counts, any expectation violations, and the DAG structure

If something fails — say a schema mismatch in the JSON files — Genie Code diagnoses the error, proposes a fix (like adjusting the schema inference or adding a CAST), and iterates until the pipeline succeeds.

Going Deeper: Python SDP with Genie Code

While SQL is the most common approach, SDP also supports Python for more complex transformation logic. The Python API uses decorators from the pyspark.pipelines module (imported as dp).

Here’s what a Python-based silver layer might look like when you need custom transformation logic:

from pyspark import pipelines as dp
from pyspark.sql.functions import col, upper, trim, when, lit

@dp.table(
    name="silver_orders_enriched",
    comment="Orders enriched with derived customer segments"
)
@dp.expect("valid_order_id", "order_id IS NOT NULL", on_violation="drop")
@dp.expect("valid_amount", "line_total > 0", on_violation="drop")
def silver_orders_enriched():
    return (
        spark.readStream.table("bronze_orders")
        .withColumn("line_total", col("quantity") * col("unit_price"))
        .withColumn("shipping_region", upper(trim(col("shipping_region"))))
        .withColumn(
            "customer_segment",
            when(col("line_total") >= 500, lit("premium"))
            .when(col("line_total") >= 100, lit("standard"))
            .otherwise(lit("basic"))
        )
        .withColumn("order_date", col("order_timestamp").cast("date"))
    )

You can ask Genie Code specifically for Python implementations:

Your prompt: “Add a Python-based silver transformation that enriches orders with a customer loyalty tier based on historical order count from the customers table in analytics.core.”

Genie Code will search your Unity Catalog for the customers table, understand its schema, and generate a Python file that joins and enriches appropriately.

Handling Change Data Capture (CDC)

One of SDP’s most powerful features is AUTO CDC, which handles change data capture with full support for out-of-order events. This is where things get genuinely hard in traditional pipelines — and trivial in SDP.

SQL example for CDC with SCD Type 2:

-- Streaming table to capture raw CDC events

CREATE OR REFRESH STREAMING TABLE customers_cdc_raw
AS SELECT * FROM STREAM read_files(
  '/Volumes/raw_data/customers_cdc/',
  format => 'json'
);

-- Cleansed CDC with expectations

CREATE OR REFRESH STREAMING TABLE customers_cdc_clean (
  CONSTRAINT valid_id EXPECT (customer_id IS NOT NULL) ON VIOLATION DROP ROW
)
AS SELECT
  customer_id,
  name,
  email,
  address,
  operation,
  operation_timestamp
FROM STREAM(customers_cdc_raw);

-- Apply CDC changes with SCD Type 2 history tracking

CREATE OR REFRESH STREAMING TABLE customers;

AUTO CDC INTO customers
FROM STREAM(customers_cdc_clean)
KEYS (customer_id)
SEQUENCE BY operation_timestamp
STORED AS SCD TYPE 2;

You can prompt Genie Code with something like:

Your prompt: “Add change data capture for customer updates from Debezium CDC events. I need SCD Type 2 to track historical changes to customer addresses.”

Genie Code understands the CDC patterns and generates the appropriate AUTO CDC declarations.

Data Quality Expectations: Your Safety Net

Expectations are SDP’s built-in data quality framework. There are three violation behaviors:


Behavior What Happens Use When
ON VIOLATION DROP ROW Invalid rows are silently dropped Tolerating messy source data
ON VIOLATION FAIL UPDATE Entire pipeline update fails Critical fields that must exist
(no action specified) Invalid rows are logged but kept Monitoring without blocking

Pro tip: Use Genie Code to add expectations iteratively. After an initial pipeline run, ask:

“Analyze the bronze_orders data and suggest quality expectations for the silver layer based on the actual data distribution.”

Genie Code can read the output datasets, profile the data, and propose expectations that make sense for your actual data — not just generic null checks.

Production Patterns and Best Practices

1. Pipeline Configuration with YAML Spec

Your pipeline project uses a YAML spec file for top-level configuration:

# pipeline.yaml
name: ecommerce_analytics
target_catalog: analytics
target_schema: ecommerce
libraries:
  - path: ./bronze_orders.sql
  - path: ./silver_orders.sql
  - path: ./gold_daily_revenue.sql
  - path: ./gold_top_products.sql
configuration:
  spark.sql.shuffle.partitions: "auto"

2. Parameterize with SET

Use SET to inject environment-specific configurations:

SET env = 'production';
SET raw_path = '/Volumes/${env}/raw_data/orders/';

CREATE OR REFRESH STREAMING TABLE bronze_orders
AS SELECT * FROM STREAM read_files(
  '${raw_path}',
  format => 'json'
);

3. Mix SQL and Python Files

A single pipeline can contain both SQL and Python source files. Use SQL for straightforward transformations and Python when you need UDFs, ML feature engineering, or complex business logic.

4. Use Genie Code for Ongoing Maintenance

Genie Code doesn’t just build pipelines — it monitors them. It can:

Triage failures when a pipeline run breaks
Investigate anomalies in data quality metrics
Handle DBR upgrades by updating deprecated syntax
Optimize resource allocation based on observed workload patterns

Ask it things like:

“The silver_orders pipeline has been failing since yesterday. Diagnose the issue.”

“Optimize the compute configuration for this pipeline — it’s running slowly on large backfills.”

Wrapping Up

The combination of SDP and Genie Code represents a genuine paradigm shift for data engineering on Databricks. SDP eliminates the boilerplate of pipeline orchestration, and Genie Code eliminates the boilerplate of writing SDP. What used to take days of manual pipeline construction can now happen in a single conversation.

The key takeaways:

Start with SDP — even without Genie Code, the declarative approach saves enormous amounts of manual orchestration code.
Use Genie Code Agent mode in the Lakeflow Pipelines Editor to plan, generate, and validate entire pipelines from natural language.
Build iteratively — start with a basic bronze-silver-gold structure, then ask Genie Code to add CDC handling, expectations, and enrichments.
Trust the loop — Genie Code’s ability to run the pipeline, read outputs, diagnose errors, and fix them autonomously is the real superpower.
Keep humans in control — every destructive action requires your approval. Genie Code proposes; you decide.

SDP and Genie Code are both generally available today at no additional cost for all Databricks customers. Open the Lakeflow Pipelines Editor, flip on Agent mode, and start talking to your data infrastructure.

Ready to get started? Check out the Databricks SDP documentation and the Genie Code guide for pipeline development.

Metric Views in AI/BI Dashboards & Genie (Part 2 of 3)

KrisJohannesen — Thu, 11 Jun 2026 11:50:26 GMT

This is part 2 of 3 in a series where I take you through working with Metric Views.

Part 1: Introduction to Metric Views
Part 2: Metric Views in AI/BI Dashboards & Genie
Part 3: Metric Views with Power BI and Tabular Editor

Introduction

In Part 1, we defined what metric views are, how you build and maintain them. Now we shift focus to how these views become consumable by business-users, via dashboards and conversational analytics. To do this, we will start by setting up a simple semantic model based on the TPCH sample data, and then get to work understanding how Metric Views work across the Databricks platform. Specifically we will cover:

Defining a Semantic Model and Metric Views using the new yaml version 1.1, which includes metadata, formatting and lots of other neat new features
How metric views can be used AI/BI dashboards
How metric views feed into Genie for natural-language analytics
Best practices, sharing, and governance aspects

Setup

To make things tangible, I have built a small Star Schema based model in Databricks based on the TPCH sample dataset. Apart from a bit of renaming, the main thing that has been set up, is definition of business keys, surrogate keys and primary/foreign key relations. The goal is to demonstrate how Metric Views work, and for that we need a model, but nothing too fancy. The model represents a simplified sales domain built around a single Fact table and its related Dimensions.

Fact table:

fact_lineitem; transactional data including quantity, price, discount, and order references.

Dimension tables:

dim_customer; customer attributes such as name, market segment, and nation.
dim_orders; order-level attributes including order date, status, and total price.
dim_part; product-level attributes (part key, name, type, and brand).
dim_supplier; supplier-level attributes (supplier key, name, nation).
dim_date; calendar table that is used as a role-playing dimension across 3 different dates on the fact.

Metric View

The metric view is designed using the new yaml version 1.1. This opens up a lot of new opportunities for metadata including comments, synonyms and data type configurations, which are useful not only for your Business Users, for context and documentation, but also for Genie which relies on metadata to better understand your logic. For more context around the semantic metadata the full documentation is available at Databricks

Source Code

The solution is created using a notebook, and a sql script, which allows you to set up the solution in your own workspace and follow along. The code is shared here on GitHub.

Consuming Metric Views

Both AI/BI Dashboards and Genie Spaces can be built directly on top of tables across your Unity Catalog. The advantage of Metric Views is, that it allows you to combine Facts and Dimensions, and define measures, relationships and metadata definitions that can be re-used across your entire domain.

AI/BI Dashboards

AI/BI Dashboards is Databricks' product which offers a native end-to-end BI experience. It is built on top of Unity Catalog, supports self-service creation of analytical datasets, visualizations, and integration with conversational analytics. When you have a Metric view defined, the dashboard layer can simply consume this, instead of consuming tables directly.

To create an AI/BI Dashboard on top of a Metric View:

In Databricks SQL workspace, navigate to Dashboards.
Create a new dashboard and go to the Data tab and select Add Data Source and select your Metric View.
Now you are ready to start the visualization part of the work; Create graphs, filters, text boxes and enjoy the existing dimensions and measures, or create new dashboard specific measures based on existing logic.

Genie

Genie is the natural-language interface inside Databricks aimed at business users: you ask “What was our revenue last quarter in EMEA?”. Genie translates your SQL, runs it, returns tabular and/or visual output. Because metric views encapsulate business logic and semantics, Genie can leverage that metadata and mappings to deliver accurate, governed answers.

The initial setup of using a Metric View in Genie is very similar to what we just saw above for AI/BI Dashboards:

In Databricks SQL workspace, navigate to Genie.
Create a new Genie Space and select your Metric View as tour data source
Now you are ready to start configuring the Genie. Provide sample questions and/or queries, teach it additional synonyms or instructions or attach additional data for cross-collaborations

SQL querying

I just want to include this here as an honorable mention. While business users may prefer interacting with Genie and Dashboards, business analysts might prefer being able to query data directly themselves, or you might want to use a Metric View to expose data to a subsequent solution outside

Metric Views are queryable directly, allowing re-use of existing Metric View definitions in subsequent data models or even building Metric Views on top of other Metric Views (... that one might get a little messy down the road). To query a metric view, the main distinction from regular SQL is the use of the MEASURE command as shown below:

SELECT 
  `Customer Region`,
  `Ship Year`,
  MEASURE(`Total Net Amount`) AS revenue,
  MEASURE(`Total Orders`) AS orders,
  MEASURE(`Average Order Value`) AS aov
FROM main.demo_tpch.order_metrics_mv
WHERE `Ship Year` = 1997
GROUP BY `Customer Region`, `Ship Year`
ORDER BY revenue DESC;

So for all that work, what do we actually get?

Building a semantic layer, is all about enabling your business users to get closer to that self-service analytics dream. By keeping the entire solution inside of Databricks, we enable the re-use of all the key governance, lineage and security features that already exist across the Data Platform. Being able to define relations, computing measures and adding additional semantics are options that we do not normally see as part of a Lakehouse. This options extends the reach of the Lakehouse platform all the way to the end-users.

Genie and Dashboards working side-by-side and in unison. Sorry for the lack of attention to the dashboard. They can actually get really pretty now with themes, colors and text, if you know what you are doing.

The key benefits of using Metric Views as the source for your solution, can be summarized to;

Consistency: all visuals and conversational queries rely on the same metric definitions.
Governance: because the metric view is under Unity Catalog governance, access controls and lineage apply.
Scalability: business users get self-service via dashboards, Genie and also SQL analytics directly on the Metric view.
Extendibility: when a new measure or dimension is added in the metric view, dashboards and Genie automatically pick it up (once surfaced in UI) without redefining logic everywhere.

The ability for end-users to query, talk to and analyze data using consistent definitions and extend these are where we get to the actual value creation of the Data Platform.

Summary and next steps

In this article we moved from definition to consumption, using the built-in Databricks tools.

In the next and final part of the series we’ll explore Part 3: Metric Views and Power BI + Tabular Editor, demonstrating how those same semantic definitions can be surfaced in external BI tools.

Databricks now supports importing Tableau and Power BI files into Genie Code to automatically build

szymon_dybczak — Sat, 23 May 2026 07:24:40 GMT

With Genie Code, you can now add a Tableau or Power BI file and have it build an AI/BI dashboard that replicates your existing visualizations - while connecting them to metric views that mirror the underlying business logic.

Import BI files using Genie Code - Azure Databricks | Microsoft Learn

Many organizations have years of BI logic embedded inside workbooks, reports, templates, and semantic layers. Rebuilding that logic manually in a new platform can be slow, error-prone, and difficult to govern.

This new workflow helps accelerate that migration path:

Upload a Tableau or Power BI file directly into Genie Code Supported formats include .twb, .twbx, .tds, .tdsx, and .pbit.
Use the /importBI command in Agent mode Genie Code imports the BI asset and generates an AI/BI dashboard.
Review the generated dashboard and metric views Measures and dimensions from the original file are transformed into metric views.
Promote metric views to Unity Catalog This makes them reusable across dashboards, Genie Spaces, and notebooks, while adding governance, lineage, access controls, and discoverability.

Currently, there is also a 100 MB limit for direct file uploads. For larger files, the recommended path is to store the file in a Unity Catalog volume and reference it directly, for example:

/importBI @/Volumes/my_catalog/my_schema/my_volume/sales_workbook.twb

Re: How Databricks Genie Turns Collaboration Tools into AI-Powered Intelligence Platforms

Hammad-Arbisoft — Fri, 08 May 2026 12:53:10 GMT

The Dashboard Era Is Ending. Conversation Is Replacing It.