Technical Blog articles

Identity Columns Best Practices for Databricks Lakehouse

LaurentLeturgez — Tue, 18 Aug 2026 11:22:44 GMT

Every data warehouse eventually needs surrogate keys, and for years that meant sequence objects, auto-increment columns, and a pile of ETL whose only job was handing out the next number. Databricks Lakehouse builds identity columns straight into Delta tables, so most of that machinery just disappears.

But they don't behave quite like the auto-increment you're used to. Assume they do, and you'll spend a frustrating afternoon working out why your row count and your maximum key value disagree.

This post is the field guide to using them well:

- GENERATED ALWAYS vs BY DEFAULT and the one-line SYNC IDENTITY fix that saves you from two rows fighting over the same key
- Why gaps are a feature, not a bug, and the three assumptions that quietly break when someone reads meaning into them
- The upsert and SCD Type 2 patterns where surrogate keys truly earn their keep including how AUTO CDC in Spark Declarative Pipelines does the expire-and-insert for you
- Where identity columns don't belong: clustering keys, high-throughput streaming, and other traps

If you're modernizing off Oracle sequences or SQL Server IDENTITY, this is also the cheapest chance you'll get to drop that technical debt for good.

Let the table do the boring work of handing out keys. Here's how to get it right.

DABs Migration Guide: Terraform to the Direct Deployment Engine

himanshu_ranjan — Tue, 18 Aug 2026 09:25:25 GMT

A guide to migrate an existing Declarative Automation Bundle from the Terraform deployment engine to the direct deployment engine. It covers the standard procedure, when bundle deployment bind is required, and five behaviours that surfaced during a customer migration, and a worked example of adopting a Genie space into the bundle.

Beyond Prompting: Production-Ready Image Generation and Visual Asset Management on Databricks

andreas-kopp_da — Mon, 17 Aug 2026 12:59:37 GMT

Generative AI for images has graduated from experimental prompts to critical business workloads. Learn how to build a unified, governed engine that brings image creation, editing, analysis, and management under a single roof.

LLMOps for Data Scientists and AI Builders: A Quickstart on Databricks

CarlosDip — Wed, 12 Aug 2026 17:09:14 GMT

Building an LLM prototype is easy. Making it reproducible, measurable, and governed is the part that takes work. This quickstart carries one support-ticket classifier through the whole lifecycle on Databricks: an agent served as an App, its LLM governed through a Unity AI Gateway model service, and an MLflow 3 evaluation gate before anything ships.

Migrate your Dashboards to AI/BI with Genie Code

pedrozanlorensi — Wed, 12 Aug 2026 11:42:41 GMT

Learn how to use Genie Code to migrate your Dashboards to Databricks AI/BI!

Triggered vs. Continuous Mode: A Deep Dive into Serverless Lakeflow Spark Declarative Pipelines

smunigati — Mon, 10 Aug 2026 18:30:31 GMT

One pipeline woke up every five minutes. The other stayed running.

Using the same e-commerce streaming workload, median end-to-end data freshness improved from 4.0 minutes to 33 seconds, while P95 latency dropped from 9.3 minutes to just 60 seconds. Those results were expected for a continuously running pipeline. What wasn't expected was what we found inside the Serverless Spark Declarative Pipeline itself: driver logs showed that eligible streaming queries could have two or more micro-batches in flight simultaneously.

That behavior is enabled by stream pipelining, a Serverless Spark Declarative Pipelines (SDP) optimization that overlaps work across successive micro-batches instead of waiting for one batch to fully complete before beginning the next. For workloads where processing time exceeds the configured trigger interval, this can significantly improve resource utilization and reduce end-to-end latency while preserving the familiar Spark Structured Streaming programming model.

What you'll learn

When to choose Triggered or Continuous mode based on workload characteristics and data freshness requirements.
How stream pipelining works internally and how it differs from the traditional sequential execution model of Spark Structured Streaming.
What we observed in production-style benchmarks and driver logs, including evidence of overlapping micro-batches and the resulting impact on latency and throughput.

The Setup

To make the comparison concrete, We built an e-commerce order-processing benchmark that represents workloads that can have both periodic analytics requirements and near-real-time operational requirements such as fraud detection and live revenue monitoring.

Producer: ~5,000 orders/min (~15,000 line items/min after explode)
Kafka: Amazon MSK cluster with TLS, connected through PrivateLink in a AWS us-west-2 region. Both Databricks and Kafka cluster running in the same region.
Compute: Serverless SDP for both pipelines using defaults settings.
Pipeline architecture at high level: Bronze → Silver → Gold
Transforms: 25+ identical transformations in the Silver layer
Test conducted during the 07/13/2026 EST timezone.

Each order contains a nested items array with 1–5 items (about three on average), so explode() amplifies the data volume by roughly 3×. The Silver layer applies SHA-256 hashes, regex extraction, multi-factor fraud scoring, state-based tax calculation, shipping-zone classification, price normalization, and other transformations.

The primary execution difference is that one pipeline runs in Triggered mode on a five-minute schedule, while the other remains active in Continuous mode and both pipelines read from the same kafka topic concurrently. The Continuous pipeline uses a 1-second trigger interval for Bronze and Silver and a 30-second interval for Gold.

Triggered vs. Continuous: The First Architectural Decision

Before looking at stream pipelining, it is important to separate two questions. The first is a pipeline lifecycle decision: should the pipeline wake up periodically, process available data, and stop—or should it remain active and continuously process new data?

Triggered Mode: Process, Stop, and Start Again

In Triggered mode, each scheduled invocation runs a pipeline lifecycle: compute is made available, the pipeline initializes, sources are connected, available data is processed, progress is committed, and the run terminates. The pipeline then waits until the next scheduled invocation and data that is continuously arrived after pipeline starts will wait for the next update.

In our benchmark, a full Triggered lifecycle took 153–194 seconds, averaging about 177 seconds per invocation. With a five-minute schedule, this created a pattern of active processing followed by idle time before the next scheduled run. In this case each cycle includes: provisioning serverless compute, JVM initialization, pipeline graph setup, Kafka consumer group creation, data processing, checkpoint commit, and teardown.

Continuous Mode: Start Once and Keep Processing

In Continuous mode, the pipeline starts and remains active. New data is processed through successive micro-batches according to the configured trigger intervals, and these trigger intervals can be configured at pipeline level or at each flow/table level, without repeating the full pipeline startup and teardown lifecycle for every scheduled processing window.

For workloads with continuously arriving data, continuous mode provides a more consistent execution model while avoiding repeated lifecycle overhead. In our benchmark, the Silver layer processed approximately 1,400 line items per batch across more than 25 transformations. The roughly 766 ms lightweight-processing time shown in the example above is intended only to illustrate how continuous mode works; it does not represent the actual Silver-layer pipeline used in the benchmark below.After the one-time startup, micro-batches fire continuously with sub-second processing times. No cold start, no teardown, no wasted compute.

When Should You Use Each Mode?

The choice should be driven by workload characteristics rather than by assuming that Continuous mode is always better.

Consideration	Triggered Mode	Continuous Mode
Data arrival	Periodic, bursty, hourly, or daily	Continuous or frequent event arrival
Freshness requirement	Minutes to hours can be acceptable	Sub-minute to few-minute freshness
Idle periods	Long idle periods between arrivals	Steady or frequent incoming data
Pipeline lifecycle	Starts and stops for each invocation	Remains active
Typical use cases	Periodic ETL, hourly fulfillment, daily reporting	Fraud detection, live dashboards, inventory tracking, CDC
Primary trade-off	Avoid unnecessary active execution during idle periods which eventually saves the execution cost.	Consistent low-latency processing and steady-state execution

Triggered mode therefore remains a valid—and often preferable—choice when data arrives infrequently or when latency is not critical. Continuous mode becomes more compelling when data arrives continuously and downstream consumers expect consistently fresh results.

Benchmark Result: The Lifecycle Choice Affects Data Freshness

We measured end-to-end data freshness as processed_at - order_timestamp: the time from when an order was created until it appeared in the enriched Silver table.

For this workload and five-minute Triggered schedule, Continuous mode delivered substantially fresher data. That result is useful, but it is not the entire story. Once Continuous mode is selected, Serverless SDP introduces another optimization worth understanding: Stream Pipelining.

Going Deeper: What Is Different Inside Continuous Mode?

This is where the second comparison begins. Triggered versus Continuous describes the pipeline lifecycle. Stream pipelining describes how eligible micro-batches can be executed within a continuously running Serverless SDP pipeline.

To understand why this matters, consider a simple question: what happens when a streaming query has a 1-second trigger interval, but each micro-batch takes more than one second to complete?

Classic Spark Structured Streaming: Sequential Micro-Batch Execution

In the traditional Spark Structured Streaming micro-batch execution pattern, a streaming query processes micro-batches sequentially. Spark still parallelizes tasks and stages within each micro-batch, but successive micro-batches of the same query do not independently execute as fully overlapping batches.

An important distinction: Spark task parallelism within a micro-batch is not the same thing as micro-batch pipelining across batches. The comparison here is specifically about whether work associated with Batch N+1 can overlap with work from Batch N.

If a micro-batch takes longer than the configured trigger interval, the next batch must wait for the current batch to finish before it can start

The above figure illustrates exactly how standard Spark Structured Streaming executes micro-batches strictly one at a time. Each micro-batch must complete all phases—Plan, Execute, Write, and Commit—before the next micro-batch can begin, even if the configured trigger interval is shorter than the batch processing time. As a result, the effective trigger interval increases from 1,000 ms to 1,450 ms, leaving the compute idle between batches and reducing overall resource utilization and throughput.

Certain resources are underutilized during phases (writing to Delta, committing checkpoints) that do not saturate the cluster, pipelining can use otherwise available capacity for a subsequent batch.

Serverless SDP: Stream-Pipelined Micro-Batch Execution

Databricks docs describe it simply:

"Instead of running microbatches sequentially like standard Spark Structured Streaming, serverless Lakeflow Spark Declarative Pipelines runs microbatches concurrently, improving compute resource utilization. Stream pipelining is enabled by default in serverless pipelines."

But what does "concurrently" actually mean? Let's visualize it.

For eligible streaming queries, Serverless Lakeflow Spark Declarative Pipelines can use stream pipelining. Instead of requiring strictly serial end-to-end completion of successive micro-batches, the engine can overlap eligible work across batches.

Conceptually, while an earlier batch is completing later work such as writing and committing, a subsequent batch can already be progressing. The pipeline remains micro-batch based; stream pipelining adds another dimension of execution concurrency.

The key insight: while batch 1 is writing data to Delta and committing the checkpoint, batch 2 is already reading from source and executing its transformations. The CPU cores that would be idle during I/O wait are now processing the next batch.

How Deep Can Pipelining Go?

Our benchmark showed that overlap was not limited to two batches. In the Silver query, we observed cases where up to three micro-batches were in flight simultaneously.

The Silver layer was configured with a 1-second trigger interval, while its average batch duration was 2,242 ms. This created sustained pressure where processing duration exceeded the configured cadence—exactly the scenario where pipelined execution becomes particularly relevant.

Going Under the Hood: What Do the Driver Logs Tell Us?

So far, we’ve seen conceptually how stream pipelining allows micro-batches to overlap. Now, let’s go one level deeper and look at what actually happens under the hood.

We analyzed approximately 230 MB of driver logs from a 30-minute Continuous-mode benchmark run to understand how Serverless SDP schedules and overlaps micro-batches during execution. SDP engine logs specific strings that encode its internal decisions and per-batch timings. The method was to grep those out and count/bucket them.

Pipelining Is Enabled at the Streaming Query Level:

The engine logs an explicit status message for each streaming query at startup:

INFO MicroBatchExecution: [queryId = c831c] Pipelined execution is enabled for query c831c9dc-073f-4d03-876f-85c7e23114d2. Reason: isServerless = true pipeliningEnabledInServerless = true isStateful = false <-- stateless = eligible deltaSinkWithCompleteMode = false <-- append mode = eligible sinkSupport = true sourcesSupport = true isPipeliningForceDisabled = false isStatefulPipeliningForceDisabled = false sameDeltaSourceSink = false

All four streaming queries in the benchmark logged pipelining as enabled, including the Gold query. Eligibility is evaluated per query.

What Micro-Batch Overlap Looks Like in Practice:

The micro-batch pipelining progress metrics give us an even deeper view into what is happening during execution. In the Silver query, 99.7% of the observed batches showed overlap. Most had two micro-batches in flight, while 6.1% showed three micro-batches in flight simultaneously.

Query	No overlap	2 in flight	3 in flight
Silver	1 (0.3%)	308 (93.6%)	20 (6.1%)
Bronze	5 (1.6%)	315 (98.4%)	0
Gold	60 (100%)	0	0
Metrics	312 (99.4%)	2 (0.6%)	0

Here's the raw log entry showing 3 concurrent batches in silver:

[queryId = c831c] [batchId = 96] Streaming query made progress: "timestamp" : "2026-07-13T02:30:59.729Z", <-- batch 96 STARTS "batchDuration" : 2270, <-- runs until 02:31:01.999Z "numInputRows" : 106 [queryId = c831c] [batchId = 97] Streaming query made progress: "timestamp" : "2026-07-13T02:31:00.729Z", <-- batch 97 STARTS (1.0s later) "batchDuration" : 2457, <-- runs until 02:31:03.186Z "numInputRows" : 394

Batch 97 begins 1.0s after batch 96, but batch 96 keeps running for another 1.27s. Their execution windows overlap by 1,270 ms: two micro-batches processing concurrently against a 1s trigger.

Batch Duration vs. Trigger Interval

Query	Trigger	Avg Batch Duration	Exceeds Trigger	Batches
Silver	1s	2,242 ms (2.2×)	100%	329
Bronze	1s	1,421 ms (1.4×)	99.7%	320
Gold	30s	7,885 ms (0.26×)	0%	60
Metrics	1s	2,387 ms (2.4×)	96.5%	314

The Silver query is the clearest example: average processing duration was more than twice the trigger interval, and nearly every observed batch showed overlap. The Gold query, by contrast, completed well within its 30-second interval and showed no observed batch overlap in this run.

Cross-Table Parallelism: The Other Dimension

Across tables: different tables in the pipeline can execute as independent streaming queries.
Within a query: eligible successive micro-batches can overlap through stream pipelining.

These dimensions are in addition to Spark's normal parallel execution of tasks within an individual micro-batch.

Which Queries Can Benefit from Pipelining?

Not every streaming query is guaranteed to use stream pipelining. The engine evaluates eligibility independently for each query.

An interesting finding in our benchmark was that the Gold query in this environment logged pipelining enabled despite state, then note it showed no actual overlap because its duration was below the trigger. It used a watermark with approx_count_distinct and append-mode output. This suggests that eligibility should be understood at the query level rather than reduced to a simple rule that all stateful queries are excluded.

The Complete Picture

The complete comparison therefore has two levels. First, choose Triggered or Continuous mode based on workload arrival patterns, latency requirements, idle time, and operational goals. Second, when Continuous mode is the right choice, Serverless SDP can provide an additional execution advantage through stream pipelining for eligible queries.

Practical Recommendations

Use Triggered Mode When:

Data arrives in infrequent or predictable bursts, such as hourly or daily loads.
There are long idle periods between data arrivals.
Processing latency is not critical and several minutes—or longer—is acceptable.
The workload is periodic ETL or reporting rather than continuous operational processing.
Avoiding active pipeline execution during long idle windows is an important consideration.

Use Continuous Mode When:

Data arrives continuously from sources such as Kafka, CDC feeds, or event streams.
Sub-minute to few-minute freshness matters.
The workload supports operational use cases such as fraud detection, live dashboards, or inventory tracking.
A steady-state pipeline is preferable to repeated startup and teardown cycles.
After Continuous is justified by the freshness SLO, eligible queries may gain additional utilization/throughput benefits from stream pipelining.

When Tuning Continuous Pipelines:

Choose trigger intervals based primarily on freshness requirements and workload characteristics—not simply to force overlap.

Warning : The reason we choose 1 second interval for Bronze and Silver tables is to showcase this overlapping, but do not shorten the trigger merely to force overlap. The success criterion is meeting the freshness SLO without sustained backlog or instability, not maximizing the number of concurrent batches.

Monitor whether batch processing duration consistently exceeds the configured trigger interval.
Use pipeline metrics and logs to understand whether pipelining is active for individual queries.
Remember that task-level Spark parallelism, cross-table parallelism, and micro-batch pipelining are separate dimensions of concurrency.
Use pipelines.trigger.interval per table to match processing complexity

Conclusion

Triggered and Continuous modes solve different problems. Triggered mode makes sense for periodic or bursty workloads where latency requirements are relaxed and the pipeline can remain inactive between processing windows. Continuous mode is better suited to continuously arriving data and workloads that require consistently fresh results.

But for Continuous workloads, Serverless Lakeflow Spark Declarative Pipelines introduce another important consideration: stream pipelining. Rather than following only the traditional sequential micro-batch execution pattern, eligible queries can overlap work across successive micro-batches, which eventually improves overall performance and throughput by using those idle CPU cycles.

In our benchmark, the Silver query averaged 2,242 ms per batch against a 1-second trigger interval. We observed overlap in 99.7% of Silver batch observations, including cases with three micro-batches in flight simultaneously improving the throughput by ~1.7X. The driver logs and micro-batch progress metrics provided direct evidence of that behavior.

The takeaway is not that Continuous mode is always better than Triggered mode. The right mode depends on the workload. But when Continuous processing is the right architectural choice, Serverless SDP's stream pipelining can provide an additional execution advantage—helping eligible streaming queries overlap micro-batch work and use available compute more effectively when processing pressure exceeds the configured trigger cadence.

How to Build, Test, and Ship SQL Pipelines on Databricks

matthew_m — Mon, 10 Aug 2026 17:35:00 GMT

Spark Declarative Pipelines (SDP) give SQL developers a first-class ETL experience. It is purpose-built for SQL, and it handles batch as well as streaming.
There is now a full dev loop for SQL pipelines: you can either author in the new Databricks pipelines editor or from your local IDE, validate with a dry run, and ship seamlessly from dev to prod environments with Declarative Automation Bundles (DABs).
Data quality and modular, environment-portable pipelines are built in. Expectations give you the row-level checks dbt users reach for, and SQL parameters let one pipeline run cleanly across dev, staging, and production.

Building a Production LangGraph Agent on Databricks - NorthStar Brand Copilot

SashankKotta — Mon, 10 Aug 2026 11:43:09 GMT

We built NorthStar Brand Copilot, an AI assistant for brand managers and field-sales reps at a fictional multi-category CPG company (Snacks, Beverages, Personal Care). It's a LangGraph custom agent that routes each question to the right Databricks-native capability:

Genie (NatualLanguage→SQL) for the numbers — sell-in/sell-out, trade-promotion ROI, inventory, market share
AI Search(RAG) for the documents — product specs, allergens, consumer reviews, brand guidelines, the promo playbook
Lakebase (long-term memory) for the decisions, For ex: "remember we cut BOGO (Buy One Get One) at Walgreens," recalled across sessions. This provides a persistent, semantic context layer, ensuring the agent doesn't process queries in isolation but evolves its understanding based on past decisions and specific user history.
LangGraph for agent orchestration. While simpler frameworks (like standard LangChain chains or monolithic agent patterns) are easier to start with, LangGraph was chosen for its superior ability to handle cyclical, complex workflows and granular state management. It allows the agent to reason, refine its tool-calling strategy iteratively, and maintain persistent state, which is essential for enterprise-grade reliability compared to more rigid, linear alternatives.

Note: The CPG organization and data used in this blog post are fictional and intended for demonstration purposes only.

Building Custom Apps on Lakehouse

CarlosDC — Fri, 07 Aug 2026 20:21:53 GMT

Plug & Play: Zerobus Ingest Now Supports Apache Kafka® Compatible APIs (Beta)

Vicky_Bukta_DB — Fri, 07 Aug 2026 17:03:34 GMT

Learn how to use your existing Apache Kafka producer clients to ingest data directly into your Delta lakehouse with new beta support for Kafka-compatible APIs.

Load Testing Databricks SQL Warehouses with JMeter — Part 3: Running and Analyzing Results

DineshBabuK — Wed, 05 Aug 2026 11:24:28 GMT

Load Testing Databricks SQL Warehouses with JMeter — Part 2: Concurrency and the Test Plan

DineshBabuK — Wed, 05 Aug 2026 11:22:49 GMT

Load Testing Databricks SQL Warehouses with JMeter — Part 1: Inputs and Configuration

DineshBabuK — Wed, 05 Aug 2026 11:20:57 GMT

[PARTNER BLOG] Evolving Metric Views: YAML, UI, Genie Code & Materialization

V_Chanakya_Raju — Wed, 05 Aug 2026 09:02:53 GMT

What if creating governed business metrics no longer meant spending most of your time writing YAML?

That's exactly the direction Databricks is moving with the evolution of Metric Views.

In this article, I explore how Metric Views have progressed from YAML-based definitions to a richer experience powered by the Metric Views UI, Genie Code, and Materialization. Through practical examples, screenshots, and real observations, I demonstrate how these capabilities simplify metric development, maintain consistency between UI and YAML, accelerate creation with AI, and improve query performance using precomputed results.

If you're interested in building a governed semantic layer in Databricks, I hope this walkthrough provides useful insights into these latest enhancements.

Custom Visualizations in AI/BI Dashboards: Build a Pareto Chart with Vega-Lite

pstyld — Mon, 03 Aug 2026 19:50:51 GMT

Need a visualization your standard AI/BI dashboard widgets can't create? Databricks AI/BI custom visualizations let you use Vega-Lite to build highly tailored charts. Learn how to craft a dynamic Pareto analysis with automated sorting, dynamic labels, background shading and Genie-assisted code enhancements.

Try the new Zerobus Rescue Column: Choose-Your-Own-Adventure Schema Management (Beta)

Vicky_Bukta_DB — Wed, 29 Jul 2026 21:11:44 GMT

It is Friday afternoon. An upstream team ships a harmless-looking change: they add a region field to the events your service already emits, and somewhere along the way, a sensor starts reporting temperature as "72F" instead of 72. You did not write that producer. You do not control its release schedule. And yet by Monday, your ingestion is either dropping those records on the floor or flat-out stalling. Every data engineer has lived some version of this, and now has nightmares reliving this data disruption.

The fundamental question when designing these pipelines is: “How strict should ingestion be about the shape of your data?”

Today, we are introducing the Zerobus Rescue Column (Beta), offering a way to answer that question in a way that makes sense for your organization. This is a designated Variant type column used to “catch” any extra or schema-mismatched fields.

Two camps, one spectrum

I have heard both arguments, and honestly, both have merit.

One camp says, "I don't want to stream garbage into my lakehouse. Gate it. If a record doesn't meet my contract, fail the producer; it's their problem."

The other camp says, "Accept everything. I would rather capture a messy record now and clean it up later than lose it forever. I don’t control the producers, but I’m responsible for ensuring data lands."

Most teams live somewhere on this spectrum, and where they sit changes based on how responsibilities are delegated across your organization. The design goal for Zerobus was never to pick a side for you. It was to let you choose and to make that choice a property of your table rather than the ingestion service, because we know you (as the data engineer) cannot change the political cards; they are simply dealt to you.

The table is the contract

Your Delta table schema is the contract. Zerobus validates every incoming record against that contract, and you decide how strict or how accepting the contract is. Think of it as a choose-your-own-adventure. The same service can enforce a rigid schema, accept a flexible subset of columns, or catch everything that does not fit, depending entirely on how you define your table.

Zerobus Ingest holds three fundamental principles:

Zerobus gates the data. It validates every record against the target table and rejects anything that does not fit. It never guesses, and it never silently drops a column. It fails fast and loud!
You define the contract. Marking columns as required or nullable and adding a rescue column are how you decide what "fits."
Zerobus never augments your table. It does not add columns, change types, or evolve your schema to accommodate a record. You evolve what Zerobus accepts by evolving the table, not the other way around, making your table the data contract.

That last principle is the one people underestimate: Zerobus is predictable by design. Its number one job is to land your data durably and predictably in the lakehouse. If it quietly reshaped your table every time a producer sent something unexpected, it would not be predictable for downstream consumers. Everything below is built on top of that guarantee.

Choose your own adventure: Schema edition

Scenario 1: Accept a subset (all columns optional)

Make every column nullable. Producers can send any subset of the columns, and anything they omit is written as NULL.

CREATE TABLE main.default.air_quality ( device_name STRING, temp INT, humidity INT);

{"device_name": "sensor-1", "temp": 22, "humidity": 55} is accepted.
- All columns present.
{"device_name": "sensor-1"} is accepted.

temp and humidity are nullable, so they are written as NULL.

{"device_name": "sensor-1", "temp": 22, "region": "us-west"} is rejected.

region does not exist in the table.

This is flexible about what you omit, but still strict about what you add.

Scenario 2: Enforce specific fields (the NOT NULL gate)

Mark columns NOT NULL to require them. Every record must supply those fields, or it is rejected. This is the strict end of the spectrum, and NOT NULL is your gate.

CREATE TABLE main.default.air_quality ( device_name STRING NOT NULL, temp INT NOT NULL, humidity INT);

{"device_name": "sensor-1", "temp": 22, "humidity": 55} is accepted.

All required columns present.

{"device_name": "sensor-1", "temp": 22} is accepted.

humidity is nullable, so it is written as NULL.

{"device_name": "sensor-1"} is rejected.

temp is required and missing.

This is the answer for the first camp. If you want to guarantee that garbage never lands, NOT NULL is the lever.

Scenario 3: Catch everything else (the rescue column)

Add a VARIANT rescue column to capture the fields that do not fit, instead of rejecting the record. Fields that match the table are written to their own columns as usual; any extra or non-conforming fields are grouped into the rescue column as a JSON object. This is the most accepting end of the spectrum, with nothing lost to rejection.

This is the answer for the second camp. It is also the new capability we are introducing today.

Rescue to the rescue

Earlier this year, we added Variant type support so you could ingest semi-structured JSON without pinning down every field in advance. The rescue column is the natural extension of that work.

The distinction is one of intent:

Variant is for when you expect semi-structured data and want to store it natively. You reach for it deliberately.
The rescue column is for when you have a schema you believe in, and you want a safety net for the strays.

You can schematize less up front and still keep that insurance policy. Anything extra a producer sends lands in the rescue column instead of being lost, and you keep the durability guarantee that matters most.

Configuring a rescue column

There is no magic column name. You designate a column as the rescue column by giving it three properties:

It allows null values.
It uses the VARIANT type.
It has the zerobus-rescue tag applied in Unity Catalog.

What lands in the rescue column, and what doesn't

Zerobus routes each field in a record based on how it fits the table:

Field in the record	Outcome
Matches a column by name and type	Written to that column normally
Not present in the table schema	Captured in the rescue column
Present in the schema, but the value's type doesn't match and the target column is nullable	Captured in the rescue column

Here is the whole thing in one example. Consider a table where rescue is nullable, uses VARIANT, and carries the zerobus-rescue tag:

CREATE TABLE main.default.air_quality ( device_name STRING NOT NULL, temp INT, humidity LONG, rescue VARIANT );

Now ingest a record with both a type mismatch and an extra field:

{
  "device_name": "sensor-1",
  "temp": "72F",
  "humidity": 87,
  "extra_field": "some value"
}

The result:

device_name	temp	humidity	rescue
sensor-1	null	87	{"extra_field": "some value", "temp": "72F"}

Both non-conforming fields, the mistyped temp and the unknown extra_field, are grouped into the rescue column's JSON object. That Friday-afternoon change from the opening? It lands cleanly, and you still have every byte to reconcile later.

Recommendations

A few things to keep in mind as you design tables around this:

Evolve the table, not the ingest. Zerobus never auto-evolves your schema. When your data shape changes permanently, evolve the table first (for example, with ALTER TABLE), then send records against the new schema.
Keep your schema backward compatible. Adding a nullable column is non-breaking, so you can roll out table and producer changes on separate timelines.
The rescue column is a landing zone, not a destination. Once data is durable, you can transform and schematize it downstream. A common pattern is to fan out: read the rescue column, promote fields that have become common into real typed columns, and route the rest onward. The rescue column buys you time to make those decisions with the data safely in hand.
Combine strictness with rescue deliberately. NOT NULL on the fields you truly require, plus a rescue column for everything else, gives you a table that both guarantees your core contract and never loses the extras. That combination is where most teams end up.

Try it out today

Our whole philosophy is the following: Zerobus makes your data durable and is reliable when it comes to landing your data. You decide how much to schematize along the way.

Your adventure, your rules.

Ready to try it? Check out the Zerobus rescue column documentation to design your contract. Be sure to enable the Beta feature first.

Have questions, or want to share how you are shaping your schemas with Zerobus? Join the discussion below!

From Experiment to Prod: LakeFlow Spark Declarative Pipelines Testing Blueprint

Taras_Chaikovsk — Tue, 28 Jul 2026 15:29:51 GMT

Part 2 of the testing blueprint: how to test SDP pipelines end-to-end. Decouple your logic for fast unit tests, then use pytest, the Databricks SDK, and Databricks Connect to run the real pipeline against ephemeral, isolated Unity Catalog schemas, all from your IDE or test notebook.

Serve Tableau Reports Directly from Databricks SQL

SepidehHashemi — Tue, 28 Jul 2026 13:29:14 GMT

Connect Tableau to Databricks SQL to eliminate data warehouse sync pipelines, cut compute costs, and serve dashboards directly from your Lakehouse.

[CUSTOMER BLOG] How S&P Global Energy Made Its Structured Data Estate Conversational with Databricks

debaprasad — Mon, 27 Jul 2026 15:26:44 GMT

From Data to Dialogue: How S&P Global Energy Made Its Structured Data Estate Conversational with Databricks Genie Agents and MCP

S&P Global Energy × Databricks — Customer Success Story

If you have ever tried to make a large, complex structured data estate available to AI agents and assistants, you have likely run into the same wall we did: agents are only as good as the context they can reach, and enterprise data rarely lives in one neat, well-documented place.

At S&P Global Energy, our data spans Chemicals, Crude Oil, Refined Products, Gas & Power, Liquified Natural Gas (LNG) and more — and each commodity is itself a rich family of datasets. LNG alone includes facility specifications, cargos , outages, supply and demand fundamentals, netbacks, historical and forecast prices, and contracts. Chemicals spans capacity, production, utilization, trade, demand by end use and by derivative, inventory change, and country- and region-level supply–demand balances. Our other commodities follow the similar patterns. This data lives across Databricks and several non-Databricks sources.

When Milliseconds Matter: Databricks Powers the Next Generation of Operational Workloads

rosinaKazakova — Mon, 27 Jul 2026 13:58:07 GMT

Databricks redefined real-time data processing with Real-Time Mode (RTM) in Spark Structured Streaming, hitting sub-300ms P99 latency for mission-critical operational workloads. Lakeflow Spark Declarative Pipelines let both pro-coders and SQL/Python developers build production streaming pipelines. The DAIS 2026 additions go further: Lakehouse//RT serves real-time queries on governed data, and Lakebase (LTAP) underpins AI agents. Together they close the loop from a millisecond shop-floor decision to coordinated, company-wide action, with no separate serving stack and no copies of data.

In this article, we discover how Databricks’ Real-Time Mode and Spark Declarative Pipelines are transforming real-time data processing in the manufacturing industry, enabling ultra-low latency for operational workloads and unifying batch and streaming analytics for AI-driven enterprises.