All blog posts in Resources

Re: Building a Production LangGraph Agent on Databricks - NorthStar Brand Copilot

Kushal_2612 — Tue, 22 Sep 2026 12:46:41 GMT

Hello, I am building a similar NL to SQL agent, but I am running into high latency with the Genie query tool (about 15 seconds per query, even simple ones).

What average response times are you getting, and how did you manage to lower them? I did really appreciate your guidance.

Re: Workflows Parameterization: Build Flexible, Production-Ready Pipelines on Databricks

supportvector_ — Mon, 21 Sep 2026 10:53:49 GMT

Amazing post!!

Multi-region model serving on Databricks with OpenSharing

KamLook — Mon, 21 Sep 2026 08:47:50 GMT

How to extend a single-region model serving stack to additional regions using serverless workspaces, OpenSharing, Real-Time Serving Endpoints, and Lakebase.

BrickTalk Recording | One Platform, Any Source: Unifying Enterprise Data with Lakeflow Connect

Tushar_Parekar — Fri, 18 Sep 2026 09:07:01 GMT

In this Community BrickTalk, @Giselle_Go_DB and @sonia_bendre demonstrate unifying enterprise data using Databricks Lakeflow Connect.

Re: 5 Ways to Enhance Databricks AI/BI Dashboard Tables with HTML

pstyld — Thu, 17 Sep 2026 15:10:19 GMT

Great question!

AI/BI Dashboard table cells configured as HTML are passed through an HTML sanitizer before rendering, which provides a platform-level safety layer (Databricks documentation).

However, the sanitizer protects the rendering step, it doesn't make values from upstream tables automatically trustworthy. The most important step is to understand where the data used to generate the HTML comes from and whether those values can be considered trusted.

If labels, URLs, or other values come from user input or come from untrusted sources, additional safeguards should be taken, such as escaping dynamic text, validating or allowlisting URLs, and avoiding the direct insertion of arbitrary data into HTML markup.

For example, in the CRM scenario from the post, if links are expected to point to a known destination such as https://example.com/crm/accounts/..., you could validate the URL against that expected link before constructing the it. If the destination itself can come from user-controlled or untrusted data and random, I'd be much more cautious about automatically turning it into a clickable link.

The right approach depends on the source data and how the HTML is constructed, but I'd treat the built-in sanitizer as defense-in-depth rather than the only security control.

Tutorial: Databricks Genie for Data Engineers and Data Scientists

DataAlchemist28 — Fri, 18 Sep 2026 08:57:28 GMT

From a table to a running app: this hands-on tutorial takes you through the full data life cycle on 696 million records. You access the data through Databricks Marketplace, run EDA with Genie Code, explore it in plain English with Genie Agents, generate a Spark Declarative Pipeline and a Lakeflow Job, and build a Databricks App. At every stage you learn how to prompt Genie, review its output, and decide where your own judgement takes over. Runs serverless on Databricks Free Edition.

Re: 5 Ways to Enhance Databricks AI/BI Dashboard Tables with HTML

hschimanski — Thu, 17 Sep 2026 13:41:40 GMT

Nice approach. But I guess you need to encode values to avoid JavaScript injection? For example if label when calling html_link function would contain „</a><a href=mypishingsite.org>Click here“. Or is there some extra security methods in HTML type cell rendering?

Re: Actually understanding Unity Catalog Managed Tables

Oleksandra — Thu, 17 Sep 2026 10:50:03 GMT

Thanks @ivanvyd! Really good point!

Infrequent readers can indeed fall outside the observation window, but what actually matters is not how often it runs but whether it's a verified reader. If you're running this report from Databricks on the latest runtime, you don't need to worry about this (the latest runtimes support new features). The two situations to be careful about are external readers or a pinned DBR version that is below the latest/supported ones.

For migration planning: flag those two kinds of infrequent readers, check them against each feature's minimum runtime before migrating, and audit what got enabled via system.storage.table_auto_upgrade_operations_history. If a reader hits an unsupported feature, you can turn the feature off, and Automatic Upgrades will not re-enable it.

I hope it helps!

Beyond the Dashboard: How Transferz Built a Truly Data-Driven Company Culture With AI/BI Genie

LPurcell — Wed, 16 Sep 2026 11:59:48 GMT

Discover how Transferz used AI/BI Genie to democratize data, achieving an 89,5% adoption rate amongst business users and 60% productivity increase amongst monthly active users through a 4-phase playbook.

5 Ways to Enhance Databricks AI/BI Dashboard Tables with HTML

pstyld — Thu, 17 Sep 2026 15:21:23 GMT

AI/BI dashboard tables can render HTML, not just text. Here's how to use reusable Unity Catalog functions to turn a plain table into visual, scannable, action-ready tables.

Re: [PARTNER BLOG] Building an Agent-Native Data Quality Manager with Databricks Apps and DQX

tmaund1704 — Mon, 14 Sep 2026 21:05:44 GMT

This is great and satisfies one of the use cases within our business. Does the AI-assisted rule generation ever send raw production row values to the LLM/model endpoint, or does it only use metadata and aggregated profiling stats from inside the Databricks workspace?

Re: Actually understanding Unity Catalog Managed Tables

ivanvyd — Mon, 14 Sep 2026 19:02:22 GMT

Thank you for the explanation, @Oleksandra! I'd be interested in an example covering infrequent readers, such as an annual reporting job that uses an otherwise active table. I believe that reader could fall outside the Auto Upgrades observation window, so guidance on accounting for it would be helpful for migration planning. Appreciate your guidance!

Re: Building Stateful Agents on Lakebase

ivanvyd — Mon, 14 Sep 2026 18:57:06 GMT

Thanks for the great walkthrough, @AlexMiller. If inventory changes while a recommendation is waiting for approval, is it checked again when the run resumes? How do you decide whether the change warrants a revised plan and another review?

Re: Identity Columns Best Practices for Databricks Lakehouse

-werners- — Mon, 14 Sep 2026 08:31:42 GMT

"Sooner or later every data warehouse needs surrogate keys."
For a classic data warehouse: yes!
For a lakehouse: optional.
If your linked source systems are solid and reliable (like ERP systems, they do still exist), the need for surrogate keys is not always there and would just add complexity (the devil is in the shuffle).
To add on this: if you want to use SKs but can't/won't use delta tables (so no auto ID):

- monotonically_increasing_id() (works on multi-node)

- zipWithIndex() (only for singlenode)

- row_number (using a window and a sort, mucho expensive)

Re: Identity Columns Best Practices for Databricks Lakehouse

LaurentLeturgez — Mon, 14 Sep 2026 06:16:54 GMT

@ravikiran7241 Generally speaking, it's not a good practice - I would even say it's an anti-pattern - to use identity columns in MV. It could generate some issues like instability on refreshes. But the main issue I see to that is on the concept of MV which are derived data, the source of truth for Ids should typically live in the dimension / facts tables, not being reinvented downstream.

So I would recommend to review that and generate Ids upstream to your MV.

Re: Tutorial: Transform your Lakeflow Connect ad data into visual and conversational analytics

Khasim_1 — Sat, 12 Sep 2026 08:27:13 GMT

Much informative one.

Re: Tutorial: Transform your Lakeflow Connect ad data into visual and conversational analytics

MinuMB — Sat, 12 Sep 2026 08:09:18 GMT

Great accelerator!

Re: Meet SDP Rewind: An undo button for your ETL pipelines

ivanvyd — Sat, 12 Sep 2026 00:42:25 GMT

This is awesome, thank you for the walkthrough (especially the watermark example)!

Since Rewind is scoped to one pipeline, could you clarify the recovery sequence for a separate downstream pipeline that has already consumed the affected rows through Delta change data feed?

Under what conditions can that consumer recover correct results by processing the RESTORE and replay changes from its existing checkpoint, and when would its output tables and operator state need separate recovery?

Thank you!

Tutorial: Transform your Lakeflow Connect ad data into visual and conversational analytics

abhishek-iyer — Fri, 11 Sep 2026 19:04:51 GMT

Learn how the Ad-Genie accelerator deploys an AI/BI dashboard and a Genie agent to harness data ingested by Lakeflow Connect’s Google Ads, Meta Ads, and TikTok Ads connectors.

Meet SDP Rewind: An undo button for your ETL pipelines

harshapasala — Tue, 15 Sep 2026 02:21:58 GMT

A bad deployment slips into your pipeline on Friday, and by Monday, it has been writing incorrect data for three days, with no clean way to roll it back. You can restore one table, but a pipeline is a graph of tables, source positions, and operator state that all moved forward together, and nothing puts them back in agreement. So most teams reprocess from the beginning, burning compute to fix a three-day mistake.

Today, we're announcing the beta of SDP Rewind, an undo button for Apache Spark™ Declarative Pipelines (SDP) that rolls the whole pipeline back to a consistent point in a single operation. You then fix your code and restart, and the pipeline picks up from the rewind point, so you pay only for the window you got wrong.

Why is this hard today?

Take that Friday deploy. The pipeline processes transactions from a landing table and has read one million of them. Over the weekend, 300,000 more transactions land and get processed incorrectly by the broken deploy, and the checkpoint advances from transaction 1,000,000 to 1,300,000. Now try to repair it with the tools you have:

Restore the tables to their Friday versions, and the incorrect data is removed, but the checkpoint still reads 1,300,000. When you resume your pipeline, it starts at transaction 1,300,001, so transactions 1,000,001 through 1,300,000 are never re-read. The weekend's data is now missing.
Reset the checkpoint to Friday instead, and you get the opposite failure. The weekend's transactions are re-read and appended to the windows that already hold them, so every weekend payment is counted twice.
Fix both at once, tables back to Friday, and the checkpoint reset to match, and a third thing could still go wrong. The gold aggregation could carry its own state like a watermark on Sunday night. Restoring the tables and offsets leaves that state untouched, so the replayed transactions arrive behind the watermark and get dropped as late.

Getting all three back to the same instant by hand, across every table and everything downstream of it, is the hard problem. So most teams do the only thing that is reliably safe: throw away the progress and reprocess from the beginning. While reprocessing from the beginning is foolproof, it is expensive and scales with your data. A three-day bug in a two-year-old pipeline costs two years of compute to fix.

What is SDP Rewind?

SDP Rewind moves all of this back together in one operation. You choose a point in time, and it rolls each table back to its version from then, resets every source position, and restores the operator state of any stateful flows, all to the same instant. Take that same Friday deploy, and one rewind removes all three problems:

The tables and the checkpoint move back together. Rewind rolls both to Friday at once, so the re-read weekend transactions land in windows that no longer hold them. Nothing is skipped, so no data goes missing, and nothing is counted twice.
Operator state moves back with them. Rewind also restores the gold aggregation's watermark to Friday, so the replayed rows arrive on time rather than behind it, and none are dropped late.

Rewind Cascades to all downstream objects in the pipeline. It does not stop at one table and state. In a single operation, it rolls back all downstream Streaming Tables and Materialized Views, each with their own data, offsets and state, to the same Friday point.

Then you fix your code and start the pipeline the way you always do. There is no separate replay command. The pipeline resumes from the rewind point, reprocesses forward using the current definition, catches up to the present, and returns to normal incremental processing. The cost of the repair is proportional to the window you rewound, not to the pipeline's lifetime.

The points you can rewind to are called rewind points. The pipeline generates them automatically, about once an hour, and keeps them for 7 days. How far back you can actually go depends on your own retention: an aggressive VACUUM, or a short retention on a Kafka topic, can shorten the window before any product limit does.

In practice, this is a single API call. A rewind is issued against the pipeline with a rewind_spec that carries two things: a point in time to return to, and the datasets to rewind. The skeleton of the call looks like this:

databricks pipelines start-update <pipeline-id> --json '{ "cause": "API_CALL", "rewind_spec": { "rewind_timestamp": "<point-in-time>", "datasets": [ { "identifier": "<catalog>.<schema>.<table>", "cascade": true } ] } }'

How it works under the hood

Rewind builds on Delta time travel. Delta already versions every table and can restore one to an earlier version, and that is the primitive Rewind uses to move each table's rows back. What Delta cannot do on its own is line up which table version goes with which source position, or coordinate that across a graph of tables. Rewind records for each table version, record the source position and operator state that produced it, then issue Delta's own RESTORE across the whole graph at once. The unit of recovery becomes the pipeline, not the table.

How a rewind actually runs

When you trigger a rewind, the pipeline walks its graph from the top down, works out the version and batch for each table, and checkpoints should return to, issues a Delta RESTORE on every table, and rewinds each checkpoint (and its operator state, where applicable) to the matching batch.

A Delta RESTORE does not delete the versions you roll back; it reapplies the old state as a new commit on top, so a table taken from version 100 to version 50 now sits at version 101 with version 50's contents. Left alone, the next run would re-read versions 51 through 100 and reprocess them, recreating the duplicate problem. Rewind hands the next run a set of skip ranges that tell it which now-phantom versions to read past, so the restart starts clean.

These steps are not a single atomic transaction, so a crash could leave the pipeline half-rewound. Two guards prevent that: RESTORE is idempotent, and the pipeline blocks ordinary updates until the rewind completes. Rewinding again to the same point converges, so a failed rewind leaves the pipeline stopped but recoverable.

Rewind in practice

We built a companion repository that stages the scenario above. It is a payments pipeline built on the medallion architecture: raw transactions land in bronze, get cleaned and converted in silver, and roll up in gold, which is a stateful five-minute windowed aggregation of settled dollars per merchant. It is small enough to read in a sitting and complete enough to break on purpose and recover for real. To follow along, clone it:

git clone https://github.com/databricks-solutions/sdp-rewind-replay

Before deploying, configure variables in databricks.yml to set the catalog and schema the tables land in, the landing table's name, and a warehouse ID (which the seeding script uses to maintain a steady rate of transaction seed). Then deploy it with Declarative Automation Bundles, which provisions the catalog, schema, landing table, pipeline, and dashboard in a single step using:

databricks bundle deploy

With the seeder feeding and the pipeline consuming, you have a live, healthy stream to break. The defect is a single wrong divisor in the silver transform, at src/pipeline.py:101. Amounts arrive as integer cents, so 7969 means $79.69, and the correct line divides by 100:

.withColumn("amount", F.col("amount_minor").cast("double") / 100) # .withColumn("amount", F.col("amount_minor").cast("double") / 10)

The two lines sit right below each other in the file, and exactly one is ever active. To introduce the bug, comment out the / 100 line, uncomment the / 10 line, and redeploy; to fix it later, swap them back.

This gets past both review and the platform. The value is still a double, so the schema is unchanged, and there is nothing for the pipeline to reject. The line still reads like an ordinary unit conversion. It is wrong in exactly the way that survives every automated check.

Nothing automated catches it, so the first place it surfaces is the dashboard that the bundle deploys. Settled dollars are plotted as bars, transaction count as a line, and normally they move together: more transactions, more money. Since the deployment, the bars spike while the line stays flat; the same payments at the same volume are valued incorrectly.

Payments Dashboard: Settled dollars jump ~10x right after the deploy, while transaction count stays flat

Recovery follows the three steps. First, deploy the corrected divisor, because replaying against the still-broken logic would only recreate the corruption. Then issue the rewind. Through the CLI, it is a single call, anchored against the UTC timestamp of the last known healthy state of the pipeline:

databricks pipelines start-update <pipeline-id> --json '{ "cause": "API_CALL", "rewind_spec": { "rewind_timestamp": "<timestamp>", "datasets": [ { "identifier": "catalog.schema.silver_payments", "cascade": true } ] } }'

When the rewind finishes, every object from silver down has been carried back to the last healthy state, aligned to the same instant. From here, you start the pipeline. It picks up where the healthy history left off and processes forward through the corrected code. When it catches up, gold holds exactly the windows it would have held if the bad deploy had never happened: the corrupted rows replaced by correct ones, the healthy history before them untouched, no duplicated windows, no duplicated payments. Exactly-once semantics hold through a stateful aggregation, across a rewind, in a continuous pipeline.

Payments Dashboard: After the code fix and Rewind, settled dollars are back in line. Only the affected windows were reprocessed; the rest of the data was left untouched.

One declarative command returns the whole pipeline to a consistent state, reprocessing only the broken window and nothing else.

Try it

SDP Rewind is entering beta across Kafka, Delta, and streaming-table sources, with stateful operators. Refer to the documentation for more about the API. The full demo, including the pipeline, dashboard, and recovery flow, is available for deployment in your own workspace: databricks-solutions/sdp-rewind-replay. Run it, break it on purpose, and hit rewind yourself.