Agree it feels unnatural, but you can hide it from the remote client with a thin encoding step right before sending, so producers keep working with normal nested JSON.A small helper that encodes only the VARIANT columns:import jsonVARIANT_COLUMNS = {...
The error "unrecognized field name 'whatever'" suggests the rescue column isn't active for that table yet, so Zerobus is still validating strictly against the schema.A few things worth checking:1. Previews exist at two levels. Besides the account con...
Quick follow-up: I reproduced the hook failure locally with the latest databricks-connect (19.2.0). foreachBatch sends its function as a PythonUDF message, but its command is just the pickled function, not the (func, return_type) tuple the hook expec...
Looking at the traceback, the failure happens inside the auto-dependencies hook (auto_dependencies/hook.py, on_execute_plan), not in your code. When the plan contains a foreachBatch function, the hook receives the bare function object and tries to in...
Great points from @balajij8 on the SDP/LDP approach. I've worked on a few Data Vault implementations, and these are the things that tend to bite when the Business Vault grows:1. Hash keys: standardize before hashing. Trim, upper-case and use a fixed ...
