Databricks Community

I would separate the engine decision from the compute-startup decision. One nuance on single-node Spark: it removes cross-machine shuffle traffic, but it doesn't remove every Exchange, local spill or startup cost.A useful comparison is the complete r...

Your follow-up points to the join input as the boundary to stabilize, rather than just the final Silver schema.If the enrichment logic permits it, parse and unpivot before the stateful join, so new attributes become rows such as entity_key, attribute...

One correctness check on the pruning suggestion: adding a target-date filter to MERGE changes which rows can match.With ON t.id = s.id AND t.event_date >= batch_min_date, an existing key whose target date is older can become unmatched and get inserte...

Hi Parth, a useful update for anyone coming back to this thread: Auto Loader now supports type widening in Public Preview on DBR 16.4+ through cloudFiles.schemaEvolutionMode = addNewColumnsWithTypeWidening.Your current addNewColumns setting still doe...

You’re reading the intended behavior correctly. Zerobus should route non-conforming fields into the rescue column instead of rejecting the record. Your error suggests the feature isn’t active on that table. Per the docs, if no column qualifies, the f...

I don’t see a way to skip the callback inside the Kafka API itself, but there are workarounds.What the docs say (page updated Oct 1, 2026):OAUTHBEARER is the only auth mechanism. There is no documented anonymous or unauthenticated mode, and I wouldn’...

Agree it feels unnatural, but you can hide it from the remote client with a thin encoding step right before sending, so producers keep working with normal nested JSON.A small helper that encodes only the VARIANT columns:import jsonVARIANT_COLUMNS = {...