cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Zerobus rescue (VARIANT) not working. No Preview Available.

DB1To3
Valued Contributor II

Has anyone tried using the rescue feature for json in zerobus-ingest?  It seems like a very significant feature, assuming the schema incoming json is even 1% outside the control of team that may be ingesting on the databricks side. There needs to be a way to ingest a json record that is slightly non-conforming.  Else you get unhelpful errors.

Here is the doc:

DB1To3_0-1792157909614.png

... but I'm not able to get this working, and there is no "preview" switches for this in my DBX account settings.

The table is configured with the NULLable, tagged column as required:

DB1To3_1-1792158060551.png



Yet whenever I submit json row data (using currently HTTP/rest), I just get an error that says zerobus doesn't approve:


Record decoder/encoder error: unrecognized field name 'whatever' at line 2 column 24. Error Code: 4044, Error State: 3.

I looked for a preview in my account settings and there is none listed (supposed to be "Beta" according to docs).
Any advice would be appreciated.  Let me know if I'm misunderstanding the intended behavior of this rescue column.

1 ACCEPTED SOLUTION

Accepted Solutions

Jen_Evangelista
New Contributor III

The error "unrecognized field name 'whatever'" suggests the rescue column isn't active for that table yet, so Zerobus is still validating strictly against the schema.

A few things worth checking:

1. Previews exist at two levels. Besides the account console, check the workspace-level Previews page (user menu > Previews) in the workspace that owns the table. Some Beta features only show up there, and some are gated and need to be enabled by your Databricks account team.

2. After it's enabled, recreate the stream/connection, since the table schema is resolved when the stream is opened.

While you wait, a pattern that avoids the problem entirely is to land the full payload in a single VARIANT column and parse it downstream (schema-on-read):

1. Landing table:

CREATE TABLE catalog.schema.events_raw (
source STRING,
payload VARIANT
);

2. Producer side, wrap the whole event as a JSON-encoded string (as the docs require for VARIANT):

import json
record = {
"source": "app-x",
"payload": json.dumps(event) # any shape, unknown fields included
}

3. Parse downstream with a Declarative Pipeline:

CREATE OR REFRESH STREAMING TABLE events_parsed AS
SELECT
payload:a::string AS a,
payload:b::int AS b,
payload:c::string AS c,
payload AS raw_payload
FROM STREAM(catalog.schema.events_raw);

Nothing gets rejected for unknown fields, new fields are already stored in raw_payload, and you just add a column to the SELECT when you need them. It also pairs with your other thread about sending VARIANT as JSON-encoded strings.

View solution in original post

3 REPLIES 3

Jen_Evangelista
New Contributor III

The error "unrecognized field name 'whatever'" suggests the rescue column isn't active for that table yet, so Zerobus is still validating strictly against the schema.

A few things worth checking:

1. Previews exist at two levels. Besides the account console, check the workspace-level Previews page (user menu > Previews) in the workspace that owns the table. Some Beta features only show up there, and some are gated and need to be enabled by your Databricks account team.

2. After it's enabled, recreate the stream/connection, since the table schema is resolved when the stream is opened.

While you wait, a pattern that avoids the problem entirely is to land the full payload in a single VARIANT column and parse it downstream (schema-on-read):

1. Landing table:

CREATE TABLE catalog.schema.events_raw (
source STRING,
payload VARIANT
);

2. Producer side, wrap the whole event as a JSON-encoded string (as the docs require for VARIANT):

import json
record = {
"source": "app-x",
"payload": json.dumps(event) # any shape, unknown fields included
}

3. Parse downstream with a Declarative Pipeline:

CREATE OR REFRESH STREAMING TABLE events_parsed AS
SELECT
payload:a::string AS a,
payload:b::int AS b,
payload:c::string AS c,
payload AS raw_payload
FROM STREAM(catalog.schema.events_raw);

Nothing gets rejected for unknown fields, new fields are already stored in raw_payload, and you just add a column to the SELECT when you need them. It also pairs with your other thread about sending VARIANT as JSON-encoded strings.

DB1To3
Valued Contributor II

Hi,

>> Besides the account console, check the workspace-level Previews page (user menu > Previews) in the workspace that owns the table

You were right.  I had to find the preview under the workspace menu, not the account settings.
Everything is working now.  Thanks for the help.

aayush_410
Contributor

Youโ€™re reading the intended behavior correctly. Zerobus should route non-conforming fields into the rescue column instead of rejecting the record. Your error suggests the feature isnโ€™t active on that table. Per the docs, if no column qualifies, the feature is inactive and Zerobus rejects non-conforming fields as usual. That matches what youโ€™re seeing.

Your description mentions nullable and tagged, but the docs list three requirements, and I canโ€™t see your table definition or screenshots. Check each one:

The column type must be VARIANT. This is the most common miss. A STRING column with the tag wonโ€™t qualify.
The column must be nullable.
The Unity Catalog column tag must be exactly zerobus-rescue. Check the spelling and that itโ€™s on the column, not the table. You can verify with: SELECT * FROM <catalog>.information_schema.column_tags WHERE table_name = โ€˜<table>โ€˜;
Other things to try:

Preview toggle: some language versions of the docs say a workspace admin must opt in on the Previews page (โ€œZerobus rescue columnโ€). The current English docs just say Beta. This setting lives under workspace settings, not account settings. Check there, or ask your Databricks account team to enable it.
Stale metadata: Zerobus may cache table schema and tags. After adding or tagging the column, wait a few minutes and open a new connection or stream before retesting.
Reserved column: donโ€™t put the rescue columnโ€™s name in your JSON payload. A non-null value for it is not allowed.
Exactly one match: if more than one column meets all three criteria, Zerobus picks one arbitrarily. Keep it to one.
Malformed JSON: the โ€œline 2 column 24โ€ detail suggests the decoder is complaining about a field name. If the feature is active and you still see this, send one clean, valid JSON object with one extra field to rule out a formatting issue.
If youโ€™ve confirmed VARIANT, nullable and the exact tag, retested with a fresh connection, and it still rejects the record, open a support ticket with Databricks. Itโ€™s a Beta feature, so thatโ€™s useful feedback for them.

Aayush Sharma