Try the new Zerobus Rescue Column: Choose-Your-Own-Adventure Schema Management (Beta)

[Vicky_Bukta_DB]

It is Friday afternoon. An upstream team ships a harmless-looking change: they add a region field to the events your service already emits, and somewhere along the way, a sensor starts reporting temperature as "72F" instead of 72. You did not write that producer. You do not control its release schedule. And yet by Monday, your ingestion is either dropping those records on the floor or flat-out stalling. Every data engineer has lived some version of this, and now has nightmares reliving this data disruption.

The fundamental question when designing these pipelines is: “How strict should ingestion be about the shape of your data?”

Today, we are introducing the Zerobus Rescue Column (Beta), offering a way to answer that question in a way that makes sense for your organization. This is a designated Variant type column used to “catch” any extra or schema-mismatched fields.

Two camps, one spectrum

I have heard both arguments, and honestly, both have merit.

One camp says, "I don't want to stream garbage into my lakehouse. Gate it. If a record doesn't meet my contract, fail the producer; it's their problem."

The other camp says, "Accept everything. I would rather capture a messy record now and clean it up later than lose it forever. I don’t control the producers, but I’m responsible for ensuring data lands."

Most teams live somewhere on this spectrum, and where they sit changes based on how responsibilities are delegated across your organization. The design goal for Zerobus was never to pick a side for you. It was to let you choose and to make that choice a property of your table rather than the ingestion service, because we know you (as the data engineer) cannot change the political cards; they are simply dealt to you.

The table is the contract

Your Delta table schema is the contract. Zerobus validates every incoming record against that contract, and you decide how strict or how accepting the contract is. Think of it as a choose-your-own-adventure. The same service can enforce a rigid schema, accept a flexible subset of columns, or catch everything that does not fit, depending entirely on how you define your table.

Zerobus Ingest holds three fundamental principles:

• Zerobus gates the data. It validates every record against the target table and rejects anything that does not fit. It never guesses, and it never silently drops a column. It fails fast and loud!
• You define the contract. Marking columns as required or nullable and adding a rescue column are how you decide what "fits."
• Zerobus never augments your table. It does not add columns, change types, or evolve your schema to accommodate a record. You evolve what Zerobus accepts by evolving the table, not the other way around, making your table the data contract.

That last principle is the one people underestimate: Zerobus is predictable by design. Its number one job is to land your data durably and predictably in the lakehouse. If it quietly reshaped your table every time a producer sent something unexpected, it would not be predictable for downstream consumers. Everything below is built on top of that guarantee.

Choose your own adventure: Schema edition

Scenario 1: Accept a subset (all columns optional)

Make every column nullable. Producers can send any subset of the columns, and anything they omit is written as NULL.

CREATE TABLE main.default.air_quality (
  device_name STRING,
  temp INT,
  humidity INT);

• {"device_name": "sensor-1", "temp": 22, "humidity": 55} is accepted.
  • All columns present.
• {"device_name": "sensor-1"} is accepted. 
• temp and humidity are nullable, so they are written as NULL.
• {"device_name": "sensor-1", "temp": 22, "region": "us-west"} is rejected. 
• region does not exist in the table.

This is flexible about what you omit, but still strict about what you add.

Scenario 2: Enforce specific fields (the NOT NULL gate)

Mark columns NOT NULL to require them. Every record must supply those fields, or it is rejected. This is the strict end of the spectrum, and NOT NULL is your gate.

CREATE TABLE main.default.air_quality (
  device_name STRING NOT NULL,
  temp INT NOT NULL,
  humidity INT);

• {"device_name": "sensor-1", "temp": 22, "humidity": 55} is accepted. 
• All required columns present.
• {"device_name": "sensor-1", "temp": 22} is accepted. 
• humidity is nullable, so it is written as NULL.
• {"device_name": "sensor-1"} is rejected. 
• temp is required and missing.

This is the answer for the first camp. If you want to guarantee that garbage never lands, NOT NULL is the lever.

Scenario 3: Catch everything else (the rescue column)

Add a VARIANT rescue column to capture the fields that do not fit, instead of rejecting the record. Fields that match the table are written to their own columns as usual; any extra or non-conforming fields are grouped into the rescue column as a JSON object. This is the most accepting end of the spectrum, with nothing lost to rejection.

This is the answer for the second camp. It is also the new capability we are introducing today.

Rescue to the rescue

Earlier this year, we added Variant type support so you could ingest semi-structured JSON without pinning down every field in advance. The rescue column is the natural extension of that work. 

The distinction is one of intent:

• Variant is for when you expect semi-structured data and want to store it natively. You reach for it deliberately.
• The rescue column is for when you have a schema you believe in, and you want a safety net for the strays. 

You can schematize less up front and still keep that insurance policy. Anything extra a producer sends lands in the rescue column instead of being lost, and you keep the durability guarantee that matters most.

Configuring a rescue column

There is no magic column name. You designate a column as the rescue column by giving it three properties:

1. It allows null values.
2. It uses the VARIANT type.
3. It has the zerobus-rescue tag applied in Unity Catalog.

What lands in the rescue column, and what doesn't

Zerobus routes each field in a record based on how it fits the table:

Field in the record	Outcome
Matches a column by name and type	Written to that column normally
Not present in the table schema	Captured in the rescue column
Present in the schema, but the value's type doesn't match and the target column is nullable	Captured in the rescue column

Here is the whole thing in one example. Consider a table where rescue is nullable, uses VARIANT, and carries the zerobus-rescue tag:

CREATE TABLE main.default.air_quality (
  device_name STRING NOT NULL,
  temp INT,
  humidity LONG,
  rescue VARIANT
);

Now ingest a record with both a type mismatch and an extra field:

{
  "device_name": "sensor-1",
  "temp": "72F",
  "humidity": 87,
  "extra_field": "some value"
}

The result:

device_name	temp	humidity	rescue
sensor-1	null	87	{"extra_field": "some value", "temp": "72F"}

Both non-conforming fields, the mistyped temp and the unknown extra_field, are grouped into the rescue column's JSON object. That Friday-afternoon change from the opening? It lands cleanly, and you still have every byte to reconcile later.

Recommendations

A few things to keep in mind as you design tables around this:

• Evolve the table, not the ingest. Zerobus never auto-evolves your schema. When your data shape changes permanently, evolve the table first (for example, with ALTER TABLE), then send records against the new schema. 
• Keep your schema backward compatible. Adding a nullable column is non-breaking, so you can roll out table and producer changes on separate timelines.
• The rescue column is a landing zone, not a destination. Once data is durable, you can transform and schematize it downstream. A common pattern is to fan out: read the rescue column, promote fields that have become common into real typed columns, and route the rest onward. The rescue column buys you time to make those decisions with the data safely in hand.
• Combine strictness with rescue deliberately. NOT NULL on the fields you truly require, plus a rescue column for everything else, gives you a table that both guarantees your core contract and never loses the extras. That combination is where most teams end up.

Try it out today

Our whole philosophy is the following: Zerobus makes your data durable and is reliable when it comes to landing your data. You decide how much to schematize along the way. 

Your adventure, your rules.

Ready to try it? Check out the Zerobus rescue column documentation to design your contract. Be sure to enable the Beta feature first.

Have questions, or want to share how you are shaping your schemas with Zerobus? Join the discussion below!