ThomazNeto
Databricks Partner

I ran some tests.


**Answer — verified at source level and across six runs, Spark 3.5.2 → 4.2.0
(evidence dossier attached):**

There are two independent rejections here sharing one error class, which is
what made this confusing:

1. `CAST(struct AS VARIANT)` is illegal for *any* struct, even fully typed —
try `CAST(named_struct('a',1,'b',CAST(NULL AS INT)) AS VARIANT)`: same
CAST_WITHOUT_SUGGESTION. That's SPARK-49443 (variant objects are unordered;
use `to_variant_object`). The message prints the field types, so a VOID field
makes it *look* VOID-specific. It isn't.

2. The VOID-specific exclusion lives in `VariantGet.checkDataType`
(variantExpressions.scala): an explicit allowlist where NullType falls into
`case _ => false`. Both CAST and `to_variant_object` route through it. At
runtime, `VariantExpressionEvalUtils.buildVariant` handles null **by value**
(`if (input == null) appendNull()`) *before* matching on type — and the type
match has no NullType branch. **Encoders are selected by type; nulls are
handled by value.** A typed NULL has an encoder whose null-guard writes
variant null (your counter-example); VOID has no encoder at all — your
inference-placeholder hypothesis is the mechanical truth.

Top-level works because `case (NullType, _) => true` in canCast is matched
*before* the variant rule: nothing is encoded; you get a SQL NULL typed
VARIANT.

**Boundary in one line:** VOID may *occupy* a VARIANT slot —
`CAST(named_struct('b',NULL) AS STRUCT<b: VARIANT>)` works, b = SQL NULL —
but may never pass *through* the variant encoder:
`to_variant_object(named_struct('b',NULL))` fails.

**Workaround:** type the field first.
`to_variant_object(CAST(s AS STRUCT<a INT, b INT>))` → `{"a":1,"b":null}`;
for arrays, `CAST(array(NULL) AS ARRAY<STRING>)::VARIANT` → `[null]`.

**Deliberate or omission?** Undocumented. One hint toward omission: CHAR and
VARCHAR get an explicit `=> false` case, NullType only hits the default — and
the encoding would be trivially well-defined (every value = variant null,
type ID 0). So the JIRA stands: *support NullType in variant construction, or
document the rationale.* For the OSS repro, use `to_variant_object`/arrays,
since OSS rejects the struct CAST form entirely.

*Attachments: evidence dossier (PDF, all six runs with cluster configs and
timestamps) + raw notebook exports for DBR 16.4 LTS (Spark 3.5.2) and DBR 19
Beta (4.2.0). Identical results everywhere tested; only DBR 15.3/15.4 remain
unverified.*

 

Thomaz A. Rossito Neto
Principal Data & AI — CI&T
thomazn@ciandt.com
linkedin.com/in/thomaz-antonio-rossito-neto