All Technical Blog posts

Re: DABs Migration Guide: Terraform to the Direct Deployment Engine

-werners- — Mon, 24 Aug 2026 07:32:59 GMT

Nice to hear it is a known issue and being worked on!

Re: DABs Migration Guide: Terraform to the Direct Deployment Engine

himanshu_ranjan — Fri, 21 Aug 2026 21:01:06 GMT

Thanks @-werners-, glad the guide helped.

Your question is a bit outside what the blog covers. What you are seeing is a different thing: how the direct engine compares the values your cluster policy fills in each time you plan or deploy.

It's a known issue and it is being worked on. A recent CLI fix made the engine leave apply_policy_default_values, data_security_mode, enable_elastic_disk, node_type_id and a few others alone, which is why you see those skipped. The ones you listed, spark_conf, spark_version, spark_env_vars, runtime_engine and custom_tags, are not handled yet, so they still show up as an update every time.

A quick workaround is to set those fields yourself in the bundle YAML. Then your config matches what the policy adds and the diff goes away. You lose a little of the benefit of the policy, but it stops the constant changes until the engine catches up.

Re: DABs Migration Guide: Terraform to the Direct Deployment Engine

-werners- — Fri, 21 Aug 2026 08:47:24 GMT

Tnx, helpful guide!

However I do notice that the direct engine still lacks some functionality.
fetching values from cluster policies apparently is only partially implemented.
I had Genie make me a nice summary :-):

Summary: After migrating from the Terraform engine to the direct engine, jobs with apply_policy_default_values: true are flagged as "changed" on every bundle plan / bundle deploy, even when no YAML changes were made. This creates an infinite update loop because the cluster policy re-injects the same values after each deploy.

Current behavior: The direct engine correctly skips some policy/platform-populated fields:

azure_attributes → "action": "skip", "reason": "managed"
data_security_mode → "action": "skip", "reason": "backend_default"
enable_elastic_disk → "action": "skip", "reason": "backend_default"
node_type_id → "action": "skip", "reason": "backend_default"

But it flags these policy-injected fields as "action": "update":

spark_env_vars
spark_version
spark_conf
custom_tags (e.g. ResourceClass: SingleNode)
runtime_engine

Expected behavior: When apply_policy_default_values: true is set on a job cluster, the engine should recognize that the policy is the source of truth for fields not declared in the bundle YAML. Fields present in remote state but absent from local config — that are injected by the referenced policy_id — should be treated as "action": "skip", "reason": "managed" (or a new reason like "policy_default"), not as removals.

Re: DABs Migration Guide: Terraform to the Direct Deployment Engine

EduardoLomonaco — Fri, 21 Aug 2026 02:20:21 GMT

Great post, I wasn't aware of the change and helps me learn about it. Will greatly benefit one of my customers!

Re: DABs Migration Guide: Terraform to the Direct Deployment Engine

himanshu_ranjan — Thu, 20 Aug 2026 17:40:22 GMT

Thanks @Armanillo_RoK for reading it closely, and glad it was useful. You've read the decoupling correctly: because the direct engine no longer waits on a Terraform provider release, the set of bundle-supported resources can grow on its own, and it now reaches infrastructure-shaped objects like catalogs and external locations.

On which layer owns what, there's actually a stated recommendation, not just "wherever ships first." The Databricks deployment guide (the IaC section) suggests Terraform for infrastructure like workspaces, networking, Unity Catalog and storage, and Bundles for data and AI workloads, where it lists jobs, pipelines, notebooks and models:
https://docs.databricks.com/aws/en/lakehouse-architecture/deployment-guide/iac

Being able to deploy catalogs and external locations from a bundle is a new capability, but it doesn't move that recommendation. By default those still sit with the platform team's Terraform. I'd only manage one from a bundle as a deliberate, agreed exception. For example, a self-contained catalog that belongs to a single workload, where the platform team has decided to leave it to that team instead of managing it centrally.

On Genie, I'd keep it in the bundle, for the reason you gave. A Genie space isn't really infrastructure, it's instructions, sample questions and trusted assets that move with the data model, so it belongs next to the pipelines it describes. Terraform provider support for Genie is an open feature request, not something available today, and even if it lands I'd let ownership decide rather than the fact that a new resource exists. A Genie space tied to a data product stays in the bundle.

On two tools managing the same object, there isn't a cross-tool safeguard. Bundles on the direct engine keep their own deployment state and Terraform keeps its own state file, and neither one knows about the other, so each can think it's authoritative. bundle deployment bind and Terraform's import are explicit ways to adopt an existing resource, not conflict detectors. So the real guardrail is ownership: one authoritative tool per object, usually split by team, with the platform team owning the Terraform repo and the data engineers owning the bundle, and the bundle referencing infra objects by name instead of redeclaring them. When you do move ownership between the two, do it deliberately and check with a plan that shows no unintended create, replace or delete. That's why I lean on bundle plan as the safe preview in the post.

Re: DABs Migration Guide: Terraform to the Direct Deployment Engine

Armanillo_RoK — Thu, 20 Aug 2026 13:22:11 GMT

Hi @himanshu_ranjan,

thanks for the heads-up!

Genie Agents (still genie_spaces as a bundle resource type) landed in bundles because the Terraform provider has no equivalent resource, and the direct engine removed the requirement to wait for a provider release. Fair enough.

But that decoupling means the set of bundle-deployable resources will now grow faster than the provider's, and it has already reached infrastructure-shaped objects, namely catalogs and external locations are direct-engine bundle resources today, and those sit squarely in most platform teams' Terraform.

So is there a stated principle for which layer owns what going forward, or does it land wherever ships first?

Specifically, if the provider eventually adds a Genie resource, is the guidance to move or to stay in bundles? My own view is stay in DABs since a Genie Agent isn't really infrastructure, it's instructions, example queries and trusted assets that change with the data model, so it belongs next to the pipelines it describes. But I'd rather know if that's the intended direction.

And for a resource both layers can manage, is there anything preventing two tools from claiming the same object, or is that purely down to team discipline?

Re: Load Testing Databricks SQL Warehouses with JMeter — Part 1: Inputs and Configuration

yit337 — Wed, 19 Aug 2026 09:51:57 GMT

Great article, very useful indeed.

Identity Columns Best Practices for Databricks Lakehouse

LaurentLeturgez — Tue, 18 Aug 2026 11:22:44 GMT

Every data warehouse eventually needs surrogate keys, and for years that meant sequence objects, auto-increment columns, and a pile of ETL whose only job was handing out the next number. Databricks Lakehouse builds identity columns straight into Delta tables, so most of that machinery just disappears.

But they don't behave quite like the auto-increment you're used to. Assume they do, and you'll spend a frustrating afternoon working out why your row count and your maximum key value disagree.

This post is the field guide to using them well:

- GENERATED ALWAYS vs BY DEFAULT and the one-line SYNC IDENTITY fix that saves you from two rows fighting over the same key
- Why gaps are a feature, not a bug, and the three assumptions that quietly break when someone reads meaning into them
- The upsert and SCD Type 2 patterns where surrogate keys truly earn their keep including how AUTO CDC in Spark Declarative Pipelines does the expire-and-insert for you
- Where identity columns don't belong: clustering keys, high-throughput streaming, and other traps

If you're modernizing off Oracle sequences or SQL Server IDENTITY, this is also the cheapest chance you'll get to drop that technical debt for good.

Let the table do the boring work of handing out keys. Here's how to get it right.

DABs Migration Guide: Terraform to the Direct Deployment Engine

himanshu_ranjan — Tue, 18 Aug 2026 09:25:25 GMT

A guide to migrate an existing Declarative Automation Bundle from the Terraform deployment engine to the direct deployment engine. It covers the standard procedure, when bundle deployment bind is required, and five behaviours that surfaced during a customer migration, and a worked example of adopting a Genie space into the bundle.

Beyond Prompting: Production-Ready Image Generation and Visual Asset Management on Databricks

andreas-kopp_da — Mon, 17 Aug 2026 12:59:37 GMT

Generative AI for images has graduated from experimental prompts to critical business workloads. Learn how to build a unified, governed engine that brings image creation, editing, analysis, and management under a single roof.

Re: Migrate your Dashboards to AI/BI with Genie Code

pedrozanlorensi — Wed, 12 Aug 2026 20:52:34 GMT

@DoTA - Great input on the Serverless Budget Policy. I recommend checking out this doc for a reference on how to manage budgets and cost controls for Genie: https://docs.databricks.com/aws/en/genie/budgets

Additionally, step 9 covers UC permissions and policies review, which is super important indeed.

LLMOps for Data Scientists and AI Builders: A Quickstart on Databricks

CarlosDip — Wed, 12 Aug 2026 17:09:14 GMT

Building an LLM prototype is easy. Making it reproducible, measurable, and governed is the part that takes work. This quickstart carries one support-ticket classifier through the whole lifecycle on Databricks: an agent served as an App, its LLM governed through a Unity AI Gateway model service, and an MLflow 3 evaluation gate before anything ships.

Re: Triggered vs. Continuous Mode: A Deep Dive into Serverless Lakeflow Spark Declarative Pipelines

DoTA — Wed, 12 Aug 2026 14:40:30 GMT

Solid benchmark — the stream pipelining section is particularly valuable since that behavior isn't obvious from the docs alone and most teams wouldn't think to grep driver logs to confirm it.

One dimension worth adding to the decision table: the cost break-even point. The table notes that triggered mode "avoids unnecessary active execution cost during idle periods," which is correct — but the comparison isn't symmetric. Continuous mode pays for steady-state compute even when no data is arriving; triggered mode pays startup overhead (153–194 seconds per invocation in your benchmark) on every cycle regardless of data volume.

The math shifts with arrival rate. At low throughput (sparse events, long quiet windows), triggered on a 5-minute schedule means startup overhead is a fixed tax on a small data payload, but the pipeline is idle 70%+ of the time — triggered wins on cost. At high continuous throughput, continuous mode's steady-state efficiency closes the cost gap quickly, especially once stream pipelining kicks in and utilizes those otherwise-idle CPU cycles.

A hybrid pattern that works well in practice: Bronze/Silver on continuous to meet the freshness SLO, Gold/reporting layers on triggered at 15–30 minute intervals. Gold is typically aggregation-heavy and latency-tolerant — no user is waiting sub-minute for a daily revenue roll-up. You preserve the freshness guarantee where it matters and avoid paying continuous compute hours on your widest, most expensive tables.

The pipelines.trigger.interval per-table config you mention at the end enables exactly this within a single pipeline — worth calling out explicitly as the primary lever for this hybrid approach.

Re: Building a Production LangGraph Agent on Databricks - NorthStar Brand Copilot

DoTA — Wed, 12 Aug 2026 14:38:43 GMT

Great end-to-end write-up — the MLflow tracing setup in particular is a pattern more teams should adopt from day one rather than bolting on later.

One thing worth extending that tracing to: per-node cost attribution. Since every ChatDatabricks call in the LangGraph graph is a separate model invocation, the context window grows with each tool result appended to the message list. In a session where the agent calls Genie + Vector Search + Lakebase in sequence, you're often paying 3–5× what a single-query baseline would suggest — because each tool output accumulates in the prompt for all subsequent steps.

MLflow spans already capture token counts per step. Adding a cost_usd attribute to each span (input_tokens * price_in + output_tokens * price_out) and grouping by session_id gives you a per-conversation cost view that's essential for tuning buffer sizes before you hit production scale.

Two optimizations that help: (1) Summarize each tool response before appending it to the graph state — passing a 50-token summary instead of a 500-token raw Genie result keeps context lean across subsequent nodes. (2) Not all nodes need Sonnet-class reasoning. The routing step (which tool to call?) is usually a simpler classification task than the final answer synthesis. Splitting that to a lighter model and reserving claude-sonnet-4-5 for the final generation step can cut per-session cost 30–50% with no visible quality regression on the routing decision itself.

Re: Migrate your Dashboards to AI/BI with Genie Code

DoTA — Wed, 12 Aug 2026 14:35:34 GMT

Great structured playbook — the 10-step framing makes this approachable for teams that are intimidated by "migration" as a project.

One thing worth calling out for larger migrations: the serverless compute Genie Code spins up during /importBI runs is billed under your workspace's serverless budget, not tied to a specific project or cost center by default. If you're migrating 30+ dashboards in parallel across multiple teams, that attribution gap adds up fast and can surprise finance at month-end.

A practical guardrail before starting: set a serverless budget policy scoped by tag or user group, so you get early signals on spend per migration workstream rather than discovering it after the fact.

On the Unity Catalog promotion step (step 7) — for teams in regulated industries (banking, healthcare), this is also where a PII scan of migrated metric views pays off. Genie Code faithfully migrates computed columns and filters, but it won't flag if a measure accidentally surfaces personal data that was previously hidden by row-level security in the original BI layer. Worth a data governance review pass before broad publication, especially if your Unity Catalog has mixed sensitivity tiers.

Migrate your Dashboards to AI/BI with Genie Code

pedrozanlorensi — Wed, 12 Aug 2026 11:42:41 GMT

Learn how to use Genie Code to migrate your Dashboards to Databricks AI/BI!

Re: Custom Visualizations in AI/BI Dashboards: Build a Pareto Chart with Vega-Lite

pstyld — Tue, 11 Aug 2026 14:12:38 GMT

Great question @hschimanski

Here is the doc for it: Filter other widgets on selection.

There are some requirements listed in the doc. Here is the updated vega lite spec from this blog that uses cross filtering.

A few key points:

params goes inside the layer with the clickable mark, not at the root of the layered spec
fields must reference the field names from your widget's Fields config
Only dimension fields work (not aggregated measures)

To test it, try this.
1. Create a new bar chart widget (or any widget you want) that uses the following:

X axis - defect_category
Y axis - SUM(defect_count)

2. Then update the vega lite in the custom viz with this:

{ "$schema": "https://vega.github.io/schema/vega-lite/v6.json", "width": "container", "height": "container", "data": { "name": "databricks_query" }, "encoding": { "x": { "field": "defect_category", "type": "nominal", "sort": { "op": "min", "field": "sort_order", "order": "ascending" }, "axis": { "title": "Defect category", "labelAngle": -40 } } }, "layer": [ { "transform": [ { "filter": "datum.pareto_group === 'Key Issue Drivers'" } ], "mark": { "type": "rect", "color": "#FF5F46", "opacity": 0.15 } }, { "params": [ { "name": "databricks_mark_selection", "select": { "type": "point", "fields": [ "defect_category" ] } } ], "mark": { "type": "bar", "width": { "band": 0.8 }, "stroke": null }, "encoding": { "y": { "field": "defect_count", "type": "quantitative", "axis": { "title": "Defect count" } }, "color": { "field": "pareto_group", "type": "nominal", "scale": { "domain": [ "Key Issue Drivers", "Minor Issues", "Cumulative %" ], "range": [ "#FF5F46", "#C4CCD6", "#1B5162" ] }, "legend": { "title": null, "orient": "top" } }, "fillOpacity": { "condition": { "param": "databricks_mark_selection", "value": 1 }, "value": 0.4 }, "stroke": { "condition": { "param": "databricks_mark_selection", "empty": false, "value": { "expr": "colors.markHighlightColor" } }, "value": null }, "strokeWidth": { "condition": { "param": "databricks_mark_selection", "empty": false, "value": 2 }, "value": 0 } } }, { "layer": [ { "mark": { "type": "area", "color": "#1B5162", "opacity": 0.05, "line": false }, "encoding": { "y": { "field": "cumulative_percent", "type": "quantitative", "axis": { "title": "Cumulative %", "orient": "right" } } } }, { "mark": { "type": "line", "point": { "size": 80, "filled": true } }, "encoding": { "y": { "field": "cumulative_percent", "type": "quantitative" }, "color": { "datum": "Cumulative %", "type": "nominal" } } }, { "transform": [ { "calculate": "round(datum.cumulative_percent) + '%'", "as": "cum_label" } ], "mark": { "type": "text", "dy": -12, "fontSize": 12, "fontWeight": "bold", "color": "#1B5162" }, "encoding": { "y": { "field": "cumulative_percent", "type": "quantitative" }, "text": { "field": "cum_label", "type": "nominal" } } }, { "mark": { "type": "rule", "color": "#0b2026", "strokeDash": [ 6, 4 ], "size": 2 }, "encoding": { "x": null, "y": { "datum": 80 } } }, { "transform": [ { "filter": "datum.defect_category === 'Label Error'" } ], "mark": { "type": "text", "text": "80% of defects", "align": "center", "baseline": "bottom", "dy": -1, "fontSize": 13, "fontWeight": "bold", "color": "#0b2026" }, "encoding": { "y": { "datum": 80 } } }, { "transform": [ { "filter": "datum.pareto_group === 'Key Issue Drivers'" }, { "joinaggregate": [ { "op": "max", "field": "sort_order", "as": "max_rank" } ] }, { "filter": "datum.sort_order === round((1 + datum.max_rank) / 2)" } ], "mark": { "type": "text", "text": "Fix these first", "align": "center", "baseline": "middle", "fontSize": 15, "fontWeight": "bold", "color": "#98102A" }, "encoding": { "y": { "datum": 92 } } }, { "transform": [ { "calculate": "datum.pareto_group === 'Key Issue Drivers' ? 1 : 0", "as": "is_kd" }, { "calculate": "datum.pareto_group === 'Key Issue Drivers' ? datum.cumulative_percent : 0", "as": "kd_cum_val" }, { "joinaggregate": [ { "op": "count", "as": "total_cats" }, { "op": "sum", "field": "is_kd", "as": "kd_count" }, { "op": "max", "field": "kd_cum_val", "as": "kd_cum" } ] }, { "filter": "datum.sort_order === round(datum.total_cats * 0.7)" }, { "calculate": "datum.kd_count + ' of ' + datum.total_cats + ' categories drive ' + round(datum.kd_cum) + '% of all defects'", "as": "summary" } ], "mark": { "type": "text", "align": "center", "baseline": "middle", "fontSize": 15, "fontWeight": "bold", "color": "#1B3139" }, "encoding": { "y": { "datum": 55 }, "text": { "field": "summary", "type": "nominal" } } } ] } ], "resolve": { "scale": { "y": "independent" } }, "config": { "autosize": { "type": "fit", "contains": "padding" } } }

Example

Re: Triggered vs. Continuous Mode: A Deep Dive into Serverless Lakeflow Spark Declarative Pipelines

Shaker_R — Mon, 10 Aug 2026 20:22:42 GMT

Looks great. It really shows your deep knowledge of the system and the scenarios needed to validate it thoroughly!

it really helps us with better understanding of different modes! TY

Triggered vs. Continuous Mode: A Deep Dive into Serverless Lakeflow Spark Declarative Pipelines

smunigati — Mon, 10 Aug 2026 18:30:31 GMT

One pipeline woke up every five minutes. The other stayed running.

Using the same e-commerce streaming workload, median end-to-end data freshness improved from 4.0 minutes to 33 seconds, while P95 latency dropped from 9.3 minutes to just 60 seconds. Those results were expected for a continuously running pipeline. What wasn't expected was what we found inside the Serverless Spark Declarative Pipeline itself: driver logs showed that eligible streaming queries could have two or more micro-batches in flight simultaneously.

That behavior is enabled by stream pipelining, a Serverless Spark Declarative Pipelines (SDP) optimization that overlaps work across successive micro-batches instead of waiting for one batch to fully complete before beginning the next. For workloads where processing time exceeds the configured trigger interval, this can significantly improve resource utilization and reduce end-to-end latency while preserving the familiar Spark Structured Streaming programming model.

What you'll learn

When to choose Triggered or Continuous mode based on workload characteristics and data freshness requirements.
How stream pipelining works internally and how it differs from the traditional sequential execution model of Spark Structured Streaming.
What we observed in production-style benchmarks and driver logs, including evidence of overlapping micro-batches and the resulting impact on latency and throughput.

The Setup

To make the comparison concrete, We built an e-commerce order-processing benchmark that represents workloads that can have both periodic analytics requirements and near-real-time operational requirements such as fraud detection and live revenue monitoring.

Producer: ~5,000 orders/min (~15,000 line items/min after explode)
Kafka: Amazon MSK cluster with TLS, connected through PrivateLink in a AWS us-west-2 region. Both Databricks and Kafka cluster running in the same region.
Compute: Serverless SDP for both pipelines using defaults settings.
Pipeline architecture at high level: Bronze → Silver → Gold
Transforms: 25+ identical transformations in the Silver layer
Test conducted during the 07/13/2026 EST timezone.

Each order contains a nested items array with 1–5 items (about three on average), so explode() amplifies the data volume by roughly 3×. The Silver layer applies SHA-256 hashes, regex extraction, multi-factor fraud scoring, state-based tax calculation, shipping-zone classification, price normalization, and other transformations.

The primary execution difference is that one pipeline runs in Triggered mode on a five-minute schedule, while the other remains active in Continuous mode and both pipelines read from the same kafka topic concurrently. The Continuous pipeline uses a 1-second trigger interval for Bronze and Silver and a 30-second interval for Gold.

Triggered vs. Continuous: The First Architectural Decision

Before looking at stream pipelining, it is important to separate two questions. The first is a pipeline lifecycle decision: should the pipeline wake up periodically, process available data, and stop—or should it remain active and continuously process new data?

Triggered Mode: Process, Stop, and Start Again

In Triggered mode, each scheduled invocation runs a pipeline lifecycle: compute is made available, the pipeline initializes, sources are connected, available data is processed, progress is committed, and the run terminates. The pipeline then waits until the next scheduled invocation and data that is continuously arrived after pipeline starts will wait for the next update.

In our benchmark, a full Triggered lifecycle took 153–194 seconds, averaging about 177 seconds per invocation. With a five-minute schedule, this created a pattern of active processing followed by idle time before the next scheduled run. In this case each cycle includes: provisioning serverless compute, JVM initialization, pipeline graph setup, Kafka consumer group creation, data processing, checkpoint commit, and teardown.

Continuous Mode: Start Once and Keep Processing

In Continuous mode, the pipeline starts and remains active. New data is processed through successive micro-batches according to the configured trigger intervals, and these trigger intervals can be configured at pipeline level or at each flow/table level, without repeating the full pipeline startup and teardown lifecycle for every scheduled processing window.

For workloads with continuously arriving data, continuous mode provides a more consistent execution model while avoiding repeated lifecycle overhead. In our benchmark, the Silver layer processed approximately 1,400 line items per batch across more than 25 transformations. The roughly 766 ms lightweight-processing time shown in the example above is intended only to illustrate how continuous mode works; it does not represent the actual Silver-layer pipeline used in the benchmark below.After the one-time startup, micro-batches fire continuously with sub-second processing times. No cold start, no teardown, no wasted compute.

When Should You Use Each Mode?

The choice should be driven by workload characteristics rather than by assuming that Continuous mode is always better.

Consideration	Triggered Mode	Continuous Mode
Data arrival	Periodic, bursty, hourly, or daily	Continuous or frequent event arrival
Freshness requirement	Minutes to hours can be acceptable	Sub-minute to few-minute freshness
Idle periods	Long idle periods between arrivals	Steady or frequent incoming data
Pipeline lifecycle	Starts and stops for each invocation	Remains active
Typical use cases	Periodic ETL, hourly fulfillment, daily reporting	Fraud detection, live dashboards, inventory tracking, CDC
Primary trade-off	Avoid unnecessary active execution during idle periods which eventually saves the execution cost.	Consistent low-latency processing and steady-state execution

Triggered mode therefore remains a valid—and often preferable—choice when data arrives infrequently or when latency is not critical. Continuous mode becomes more compelling when data arrives continuously and downstream consumers expect consistently fresh results.

Benchmark Result: The Lifecycle Choice Affects Data Freshness

We measured end-to-end data freshness as processed_at - order_timestamp: the time from when an order was created until it appeared in the enriched Silver table.

For this workload and five-minute Triggered schedule, Continuous mode delivered substantially fresher data. That result is useful, but it is not the entire story. Once Continuous mode is selected, Serverless SDP introduces another optimization worth understanding: Stream Pipelining.

Going Deeper: What Is Different Inside Continuous Mode?

This is where the second comparison begins. Triggered versus Continuous describes the pipeline lifecycle. Stream pipelining describes how eligible micro-batches can be executed within a continuously running Serverless SDP pipeline.

To understand why this matters, consider a simple question: what happens when a streaming query has a 1-second trigger interval, but each micro-batch takes more than one second to complete?

Classic Spark Structured Streaming: Sequential Micro-Batch Execution

In the traditional Spark Structured Streaming micro-batch execution pattern, a streaming query processes micro-batches sequentially. Spark still parallelizes tasks and stages within each micro-batch, but successive micro-batches of the same query do not independently execute as fully overlapping batches.

An important distinction: Spark task parallelism within a micro-batch is not the same thing as micro-batch pipelining across batches. The comparison here is specifically about whether work associated with Batch N+1 can overlap with work from Batch N.

If a micro-batch takes longer than the configured trigger interval, the next batch must wait for the current batch to finish before it can start

The above figure illustrates exactly how standard Spark Structured Streaming executes micro-batches strictly one at a time. Each micro-batch must complete all phases—Plan, Execute, Write, and Commit—before the next micro-batch can begin, even if the configured trigger interval is shorter than the batch processing time. As a result, the effective trigger interval increases from 1,000 ms to 1,450 ms, leaving the compute idle between batches and reducing overall resource utilization and throughput.

Certain resources are underutilized during phases (writing to Delta, committing checkpoints) that do not saturate the cluster, pipelining can use otherwise available capacity for a subsequent batch.

Serverless SDP: Stream-Pipelined Micro-Batch Execution

Databricks docs describe it simply:

"Instead of running microbatches sequentially like standard Spark Structured Streaming, serverless Lakeflow Spark Declarative Pipelines runs microbatches concurrently, improving compute resource utilization. Stream pipelining is enabled by default in serverless pipelines."

But what does "concurrently" actually mean? Let's visualize it.

For eligible streaming queries, Serverless Lakeflow Spark Declarative Pipelines can use stream pipelining. Instead of requiring strictly serial end-to-end completion of successive micro-batches, the engine can overlap eligible work across batches.

Conceptually, while an earlier batch is completing later work such as writing and committing, a subsequent batch can already be progressing. The pipeline remains micro-batch based; stream pipelining adds another dimension of execution concurrency.

The key insight: while batch 1 is writing data to Delta and committing the checkpoint, batch 2 is already reading from source and executing its transformations. The CPU cores that would be idle during I/O wait are now processing the next batch.

How Deep Can Pipelining Go?

Our benchmark showed that overlap was not limited to two batches. In the Silver query, we observed cases where up to three micro-batches were in flight simultaneously.

The Silver layer was configured with a 1-second trigger interval, while its average batch duration was 2,242 ms. This created sustained pressure where processing duration exceeded the configured cadence—exactly the scenario where pipelined execution becomes particularly relevant.

Going Under the Hood: What Do the Driver Logs Tell Us?

So far, we’ve seen conceptually how stream pipelining allows micro-batches to overlap. Now, let’s go one level deeper and look at what actually happens under the hood.

We analyzed approximately 230 MB of driver logs from a 30-minute Continuous-mode benchmark run to understand how Serverless SDP schedules and overlaps micro-batches during execution. SDP engine logs specific strings that encode its internal decisions and per-batch timings. The method was to grep those out and count/bucket them.

Pipelining Is Enabled at the Streaming Query Level:

The engine logs an explicit status message for each streaming query at startup:

INFO MicroBatchExecution: [queryId = c831c] Pipelined execution is enabled for query c831c9dc-073f-4d03-876f-85c7e23114d2. Reason: isServerless = true pipeliningEnabledInServerless = true isStateful = false <-- stateless = eligible deltaSinkWithCompleteMode = false <-- append mode = eligible sinkSupport = true sourcesSupport = true isPipeliningForceDisabled = false isStatefulPipeliningForceDisabled = false sameDeltaSourceSink = false

All four streaming queries in the benchmark logged pipelining as enabled, including the Gold query. Eligibility is evaluated per query.

What Micro-Batch Overlap Looks Like in Practice:

The micro-batch pipelining progress metrics give us an even deeper view into what is happening during execution. In the Silver query, 99.7% of the observed batches showed overlap. Most had two micro-batches in flight, while 6.1% showed three micro-batches in flight simultaneously.

Query	No overlap	2 in flight	3 in flight
Silver	1 (0.3%)	308 (93.6%)	20 (6.1%)
Bronze	5 (1.6%)	315 (98.4%)	0
Gold	60 (100%)	0	0
Metrics	312 (99.4%)	2 (0.6%)	0

Here's the raw log entry showing 3 concurrent batches in silver:

[queryId = c831c] [batchId = 96] Streaming query made progress: "timestamp" : "2026-07-13T02:30:59.729Z", <-- batch 96 STARTS "batchDuration" : 2270, <-- runs until 02:31:01.999Z "numInputRows" : 106 [queryId = c831c] [batchId = 97] Streaming query made progress: "timestamp" : "2026-07-13T02:31:00.729Z", <-- batch 97 STARTS (1.0s later) "batchDuration" : 2457, <-- runs until 02:31:03.186Z "numInputRows" : 394

Batch 97 begins 1.0s after batch 96, but batch 96 keeps running for another 1.27s. Their execution windows overlap by 1,270 ms: two micro-batches processing concurrently against a 1s trigger.

Batch Duration vs. Trigger Interval

Query	Trigger	Avg Batch Duration	Exceeds Trigger	Batches
Silver	1s	2,242 ms (2.2×)	100%	329
Bronze	1s	1,421 ms (1.4×)	99.7%	320
Gold	30s	7,885 ms (0.26×)	0%	60
Metrics	1s	2,387 ms (2.4×)	96.5%	314

The Silver query is the clearest example: average processing duration was more than twice the trigger interval, and nearly every observed batch showed overlap. The Gold query, by contrast, completed well within its 30-second interval and showed no observed batch overlap in this run.

Cross-Table Parallelism: The Other Dimension

Across tables: different tables in the pipeline can execute as independent streaming queries.
Within a query: eligible successive micro-batches can overlap through stream pipelining.

These dimensions are in addition to Spark's normal parallel execution of tasks within an individual micro-batch.

Which Queries Can Benefit from Pipelining?

Not every streaming query is guaranteed to use stream pipelining. The engine evaluates eligibility independently for each query.

An interesting finding in our benchmark was that the Gold query in this environment logged pipelining enabled despite state, then note it showed no actual overlap because its duration was below the trigger. It used a watermark with approx_count_distinct and append-mode output. This suggests that eligibility should be understood at the query level rather than reduced to a simple rule that all stateful queries are excluded.

The Complete Picture

The complete comparison therefore has two levels. First, choose Triggered or Continuous mode based on workload arrival patterns, latency requirements, idle time, and operational goals. Second, when Continuous mode is the right choice, Serverless SDP can provide an additional execution advantage through stream pipelining for eligible queries.

Practical Recommendations

Use Triggered Mode When:

Data arrives in infrequent or predictable bursts, such as hourly or daily loads.
There are long idle periods between data arrivals.
Processing latency is not critical and several minutes—or longer—is acceptable.
The workload is periodic ETL or reporting rather than continuous operational processing.
Avoiding active pipeline execution during long idle windows is an important consideration.

Use Continuous Mode When:

Data arrives continuously from sources such as Kafka, CDC feeds, or event streams.
Sub-minute to few-minute freshness matters.
The workload supports operational use cases such as fraud detection, live dashboards, or inventory tracking.
A steady-state pipeline is preferable to repeated startup and teardown cycles.
After Continuous is justified by the freshness SLO, eligible queries may gain additional utilization/throughput benefits from stream pipelining.

When Tuning Continuous Pipelines:

Choose trigger intervals based primarily on freshness requirements and workload characteristics—not simply to force overlap.

Warning : The reason we choose 1 second interval for Bronze and Silver tables is to showcase this overlapping, but do not shorten the trigger merely to force overlap. The success criterion is meeting the freshness SLO without sustained backlog or instability, not maximizing the number of concurrent batches.

Monitor whether batch processing duration consistently exceeds the configured trigger interval.
Use pipeline metrics and logs to understand whether pipelining is active for individual queries.
Remember that task-level Spark parallelism, cross-table parallelism, and micro-batch pipelining are separate dimensions of concurrency.
Use pipelines.trigger.interval per table to match processing complexity

Conclusion

Triggered and Continuous modes solve different problems. Triggered mode makes sense for periodic or bursty workloads where latency requirements are relaxed and the pipeline can remain inactive between processing windows. Continuous mode is better suited to continuously arriving data and workloads that require consistently fresh results.

But for Continuous workloads, Serverless Lakeflow Spark Declarative Pipelines introduce another important consideration: stream pipelining. Rather than following only the traditional sequential micro-batch execution pattern, eligible queries can overlap work across successive micro-batches, which eventually improves overall performance and throughput by using those idle CPU cycles.

In our benchmark, the Silver query averaged 2,242 ms per batch against a 1-second trigger interval. We observed overlap in 99.7% of Silver batch observations, including cases with three micro-batches in flight simultaneously improving the throughput by ~1.7X. The driver logs and micro-batch progress metrics provided direct evidence of that behavior.

The takeaway is not that Continuous mode is always better than Triggered mode. The right mode depends on the workload. But when Continuous processing is the right architectural choice, Serverless SDP's stream pipelining can provide an additional execution advantage—helping eligible streaming queries overlap micro-batch work and use available compute more effectively when processing pressure exceeds the configured trigger cadence.

How to Build, Test, and Ship SQL Pipelines on Databricks

matthew_m — Mon, 10 Aug 2026 17:35:00 GMT

Spark Declarative Pipelines (SDP) give SQL developers a first-class ETL experience. It is purpose-built for SQL, and it handles batch as well as streaming.
There is now a full dev loop for SQL pipelines: you can either author in the new Databricks pipelines editor or from your local IDE, validate with a dry run, and ship seamlessly from dev to prod environments with Declarative Automation Bundles (DABs).
Data quality and modular, environment-portable pipelines are built in. Expectations give you the row-level checks dbt users reach for, and SQL parameters let one pipeline run cleanly across dev, staging, and production.