On (1): the docs don't gate Scala foreachBatch on serverless. Limitations with Databricks Connect for Scala lists streaming foreachBatch as unavailable only on Databricks Runtime 13.3 LTS and below, and serverless limitations names only Trigger.Proce...
The pattern is documented and you are right not to full refresh pipeline 2, but as written the deletion detection can never fire, and the full refresh is not why.
spark.readStream.table("raw.contact") makes contact append-only, so no key ever leaves ...
Dropping that config was the right call, and the lever you actually want is execution mode rather than a setting.
1. Flow concurrency. The properties reference is the list of supported pipeline settings, and there is no concurrency cap on it, which i...
By the time to_variant_object runs, that array has no VOID element left in it.
ARRAY(PAYMENT_DATE, TRANSACTION_DATE) resolves a common element type across its arguments, so the expression is already ARRAY<DATE>. The check walks array element types, m...
Your screenshot already uses TO_VARIANT_OBJECT; the VOID field inside the STRUCT is what causes the failure.
TO_VARIANT_OBJECT requires every field to be convertible to VARIANT, but VOID is not accepted in that nested path. Scalar CAST(PAYMENT_DATE A...
