a month ago
Has anyone tried using the rescue feature for json in zerobus-ingest? It seems like a very significant feature, assuming the schema incoming json is even 1% outside the control of team that may be ingesting on the databricks side. There needs to be a way to ingest a json record that is slightly non-conforming. Else you get unhelpful errors.
Here is the doc:
... but I'm not able to get this working, and there is no "preview" switches for this in my DBX account settings.
The table is configured with the NULLable, tagged column as required:
Yet whenever I submit json row data (using currently HTTP/rest), I just get an error that says zerobus doesn't approve:
Record decoder/encoder error: unrecognized field name 'whatever' at line 2 column 24. Error Code: 4044, Error State: 3.
I looked for a preview in my account settings and there is none listed (supposed to be "Beta" according to docs).
Any advice would be appreciated. Let me know if I'm misunderstanding the intended behavior of this rescue column.
4 weeks ago
The error "unrecognized field name 'whatever'" suggests the rescue column isn't active for that table yet, so Zerobus is still validating strictly against the schema.
A few things worth checking:
1. Previews exist at two levels. Besides the account console, check the workspace-level Previews page (user menu > Previews) in the workspace that owns the table. Some Beta features only show up there, and some are gated and need to be enabled by your Databricks account team.
2. After it's enabled, recreate the stream/connection, since the table schema is resolved when the stream is opened.
While you wait, a pattern that avoids the problem entirely is to land the full payload in a single VARIANT column and parse it downstream (schema-on-read):
1. Landing table:
CREATE TABLE catalog.schema.events_raw (
source STRING,
payload VARIANT
);
2. Producer side, wrap the whole event as a JSON-encoded string (as the docs require for VARIANT):
import json
record = {
"source": "app-x",
"payload": json.dumps(event) # any shape, unknown fields included
}
3. Parse downstream with a Declarative Pipeline:
CREATE OR REFRESH STREAMING TABLE events_parsed AS
SELECT
payload:a::string AS a,
payload:b::int AS b,
payload:c::string AS c,
payload AS raw_payload
FROM STREAM(catalog.schema.events_raw);
Nothing gets rejected for unknown fields, new fields are already stored in raw_payload, and you just add a column to the SELECT when you need them. It also pairs with your other thread about sending VARIANT as JSON-encoded strings.
4 weeks ago
The error "unrecognized field name 'whatever'" suggests the rescue column isn't active for that table yet, so Zerobus is still validating strictly against the schema.
A few things worth checking:
1. Previews exist at two levels. Besides the account console, check the workspace-level Previews page (user menu > Previews) in the workspace that owns the table. Some Beta features only show up there, and some are gated and need to be enabled by your Databricks account team.
2. After it's enabled, recreate the stream/connection, since the table schema is resolved when the stream is opened.
While you wait, a pattern that avoids the problem entirely is to land the full payload in a single VARIANT column and parse it downstream (schema-on-read):
1. Landing table:
CREATE TABLE catalog.schema.events_raw (
source STRING,
payload VARIANT
);
2. Producer side, wrap the whole event as a JSON-encoded string (as the docs require for VARIANT):
import json
record = {
"source": "app-x",
"payload": json.dumps(event) # any shape, unknown fields included
}
3. Parse downstream with a Declarative Pipeline:
CREATE OR REFRESH STREAMING TABLE events_parsed AS
SELECT
payload:a::string AS a,
payload:b::int AS b,
payload:c::string AS c,
payload AS raw_payload
FROM STREAM(catalog.schema.events_raw);
Nothing gets rejected for unknown fields, new fields are already stored in raw_payload, and you just add a column to the SELECT when you need them. It also pairs with your other thread about sending VARIANT as JSON-encoded strings.
4 weeks ago
Hi,
>> Besides the account console, check the workspace-level Previews page (user menu > Previews) in the workspace that owns the table
You were right. I had to find the preview under the workspace menu, not the account settings.
Everything is working now. Thanks for the help.
4 weeks ago
Youโre reading the intended behavior correctly. Zerobus should route non-conforming fields into the rescue column instead of rejecting the record. Your error suggests the feature isnโt active on that table. Per the docs, if no column qualifies, the feature is inactive and Zerobus rejects non-conforming fields as usual. That matches what youโre seeing.
Your description mentions nullable and tagged, but the docs list three requirements, and I canโt see your table definition or screenshots. Check each one:
The column type must be VARIANT. This is the most common miss. A STRING column with the tag wonโt qualify.
The column must be nullable.
The Unity Catalog column tag must be exactly zerobus-rescue. Check the spelling and that itโs on the column, not the table. You can verify with: SELECT * FROM <catalog>.information_schema.column_tags WHERE table_name = โ<table>โ;
Other things to try:
Preview toggle: some language versions of the docs say a workspace admin must opt in on the Previews page (โZerobus rescue columnโ). The current English docs just say Beta. This setting lives under workspace settings, not account settings. Check there, or ask your Databricks account team to enable it.
Stale metadata: Zerobus may cache table schema and tags. After adding or tagging the column, wait a few minutes and open a new connection or stream before retesting.
Reserved column: donโt put the rescue columnโs name in your JSON payload. A non-null value for it is not allowed.
Exactly one match: if more than one column meets all three criteria, Zerobus picks one arbitrarily. Keep it to one.
Malformed JSON: the โline 2 column 24โ detail suggests the decoder is complaining about a field name. If the feature is active and you still see this, send one clean, valid JSON object with one extra field to rule out a formatting issue.
If youโve confirmed VARIANT, nullable and the exact tag, retested with a fresh connection, and it still rejects the record, open a support ticket with Databricks. Itโs a Beta feature, so thatโs useful feedback for them.