- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
a week ago
I ran some tests.
**Answer — verified at source level and across six runs, Spark 3.5.2 → 4.2.0
(evidence dossier attached):**
There are two independent rejections here sharing one error class, which is
what made this confusing:
1. `CAST(struct AS VARIANT)` is illegal for *any* struct, even fully typed —
try `CAST(named_struct('a',1,'b',CAST(NULL AS INT)) AS VARIANT)`: same
CAST_WITHOUT_SUGGESTION. That's SPARK-49443 (variant objects are unordered;
use `to_variant_object`). The message prints the field types, so a VOID field
makes it *look* VOID-specific. It isn't.
2. The VOID-specific exclusion lives in `VariantGet.checkDataType`
(variantExpressions.scala): an explicit allowlist where NullType falls into
`case _ => false`. Both CAST and `to_variant_object` route through it. At
runtime, `VariantExpressionEvalUtils.buildVariant` handles null **by value**
(`if (input == null) appendNull()`) *before* matching on type — and the type
match has no NullType branch. **Encoders are selected by type; nulls are
handled by value.** A typed NULL has an encoder whose null-guard writes
variant null (your counter-example); VOID has no encoder at all — your
inference-placeholder hypothesis is the mechanical truth.
Top-level works because `case (NullType, _) => true` in canCast is matched
*before* the variant rule: nothing is encoded; you get a SQL NULL typed
VARIANT.
**Boundary in one line:** VOID may *occupy* a VARIANT slot —
`CAST(named_struct('b',NULL) AS STRUCT<b: VARIANT>)` works, b = SQL NULL —
but may never pass *through* the variant encoder:
`to_variant_object(named_struct('b',NULL))` fails.
**Workaround:** type the field first.
`to_variant_object(CAST(s AS STRUCT<a INT, b INT>))` → `{"a":1,"b":null}`;
for arrays, `CAST(array(NULL) AS ARRAY<STRING>)::VARIANT` → `[null]`.
**Deliberate or omission?** Undocumented. One hint toward omission: CHAR and
VARCHAR get an explicit `=> false` case, NullType only hits the default — and
the encoding would be trivially well-defined (every value = variant null,
type ID 0). So the JIRA stands: *support NullType in variant construction, or
document the rationale.* For the OSS repro, use `to_variant_object`/arrays,
since OSS rejects the struct CAST form entirely.
*Attachments: evidence dossier (PDF, all six runs with cluster configs and
timestamps) + raw notebook exports for DBR 16.4 LTS (Spark 3.5.2) and DBR 19
Beta (4.2.0). Identical results everywhere tested; only DBR 15.3/15.4 remain
unverified.*
Principal Data & AI — CI&T
thomazn@ciandt.com
linkedin.com/in/thomaz-antonio-rossito-neto