(Design Decision) Why VOID in STRUCT is not castable to VARIANT?

[Dhruv-22]

I asked a question recently about VOID in STRUCT failing to cast to VARIANT, whereas a VOID column is castable to VARIANT. Link

Earlier I thought it might be an error since cast was possible somehow. But the community members replied that this is the expected behaviour and not an error.

I will post an example here.

%sql
CREATE OR REPLACE TEMPORARY VIEW v_temp AS
VALUES (NULL, DATE'2025-12-31'), (NULL, DATE'2026-04-12'), (NULL, DATE'2026-06-23') AS T(PAYMENT_DATE, TRANSACTION_DATE);

SELECT PAYMENT_DATE, TYPEOF(PAYMENT_DATE), TRANSACTION_DATE, TYPEOF(TRANSACTION_DATE)
FROM v_temp;

I checked the variant encoding docs in parquet. And VOID is a type in variant. Link

What is the design decision that led to VOID being excluded from STRUCT columns for casting to VARIANT?

Kindly note that I want to know the design decision or rationale behind this. I don't need any way arounds this behaviour. I know this is the implementation and am interested in knowing why it was designed so?

[ThomazNeto]

Great question, and you're right that it's not a storage limitation — the variant encoding does have a null type (ID 0), as your Parquet link shows. The answer lives in Spark's type system, and there's actual evidence in the source and JIRAs. Fair warning: no design doc states this in one sentence, so what follows is reconstructed from the code and the tickets — but the pieces fit tightly.

Part 1: the top-level cast doesn't really "work" — it's vacuous. Spark's generic rule allows NullType → any type at analysis, and at runtime the nullSafeEval guard short-circuits: NULL in, NULL out, the cast function never executes. There's literally a comment in Cast.scala saying primitive-level casts never reach the NullType branch because of that guard — only *nested* null-type fields inside a struct reach it, and that branch throws cannotCastFromNullTypeError. So CAST(void_col AS VARIANT) never encodes a variant null; it hands you a SQL NULL that happens to be typed VARIANT. Your screenshot shows exactly that: plain nulls, no variant value ever built.

Part 2: the nested case is rejected because there, Spark would have to actually encode something — and that forces a choice it deliberately refuses to make: conflating SQL NULL with variant null. Those are different things in the variant model:

SELECT parse_json('null') IS NULL; -- false: a non-null VARIANT holding a variant-null
SELECT CAST(NULL AS VARIANT) IS NULL; -- true: a SQL NULL, nothing was encoded
SELECT to_json(parse_json('null')); -- 'null'
SELECT to_json(CAST(NULL AS VARIANT)); -- NULL

SPARK-51576 shows the project actively guarding this boundary: variant is the only type where a non-null value can cast into a null value (parse_json('null')::string → NULL), and the cast rules were tightened precisely because that conflation leaks. Now look at your struct: every value in a VOID field is a SQL NULL — typed absence. Encoding it into a variant object would require writing variant null, i.e. manufacturing "a value that says null" out of "no value." Round-trip that back out and you can no longer tell which one you started with. Rather than pick a lossy convention, the cast is rejected at analysis — hence CAST_WITHOUT_SUGGESTION, an error class that means "there is no correct suggestion to offer."

There's also a longer historical thread: NullType/VOID in Spark is a type-inference placeholder (so that literal NULLs type-check), not a materializable type. It's been progressively banned from contexts that persist or construct values — you can't create tables with VOID columns, and nested null-type casts used to crash outright with a MatchError (SPARK-27671) before becoming a proper error. Excluding it from variant *construction* while letting the vacuous top-level cast pass through is consistent with that stance: the asymmetry you found isn't VOID being partially supported in variant — it's that the top-level case never touches variant at all.

So the design decision, best reconstructed: (1) SQL NULL ≠ variant null, and casts must not silently convert one into the other; (2) VOID is a placeholder type that never materializes into constructed values. Top-level slips through because nothing is constructed; nested fails because something would have to be.

[Dhruv-22]

Hi @ThomazNeto 

Thanks for the pointers — the SPARK-51576 and the context was genuinely useful for chasing this down further.

However, there are few points I would like to bring up

• Your 'conflating SQL NULL with variant null' and 'manufacturing "a value that says null" out of "no value"' don't hold true. If I put a SQL NULL value in a typed column let's say INT, I'm able to generate a VARIANT NULL. Here is the code to prove my point
  

%sql
WITH struct_data AS (
    SELECT to_variant_object(named_struct('a', 1, 'b', CAST(NULL AS INT))) AS v
)
SELECT 
    to_json(v) AS whole_object_json,
    variant_get(v, '$.b') AS extracted_b,
    is_variant_null(variant_get(v, '$.b')) AS is_it_variant_null,
    variant_get(v, '$.b') IS NULL AS is_it_sql_null
FROM struct_data

• The roundtrip has already been addressed in spark. It can be seen in your example as well. 'parse_json('null')::string' resolves to SQL NULL instead of 'null' string.
• The only issue could have been materialization while writing as spark has an issue with VOID fields while writing. But in the following code you can see that we can easily write a variant column containing nulls.
  

To me, it seems like there is no issue in writing variant nulls and implementation already supports casting typed SQL NULLs to VARIANT NULLs. As well as, we can easily convert from VARIANT NULLs to SQL NULLs. The developers can easily implement the case where VOID columns are converted. The only reason they might not to is is because of type inference issues, typed NULLs atleast have their types known, but VOID columns would not have any datatype.

[ThomazNeto]

I ran some tests.

**Answer — verified at source level and across six runs, Spark 3.5.2 → 4.2.0
(evidence dossier attached):**

There are two independent rejections here sharing one error class, which is
what made this confusing:

1. `CAST(struct AS VARIANT)` is illegal for *any* struct, even fully typed —
try `CAST(named_struct('a',1,'b',CAST(NULL AS INT)) AS VARIANT)`: same
CAST_WITHOUT_SUGGESTION. That's SPARK-49443 (variant objects are unordered;
use `to_variant_object`). The message prints the field types, so a VOID field
makes it *look* VOID-specific. It isn't.

2. The VOID-specific exclusion lives in `VariantGet.checkDataType`
(variantExpressions.scala): an explicit allowlist where NullType falls into
`case _ => false`. Both CAST and `to_variant_object` route through it. At
runtime, `VariantExpressionEvalUtils.buildVariant` handles null **by value**
(`if (input == null) appendNull()`) *before* matching on type — and the type
match has no NullType branch. **Encoders are selected by type; nulls are
handled by value.** A typed NULL has an encoder whose null-guard writes
variant null (your counter-example); VOID has no encoder at all — your
inference-placeholder hypothesis is the mechanical truth.

Top-level works because `case (NullType, _) => true` in canCast is matched
*before* the variant rule: nothing is encoded; you get a SQL NULL typed
VARIANT.

**Boundary in one line:** VOID may *occupy* a VARIANT slot —
`CAST(named_struct('b',NULL) AS STRUCT<b: VARIANT>)` works, b = SQL NULL —
but may never pass *through* the variant encoder:
`to_variant_object(named_struct('b',NULL))` fails.

**Workaround:** type the field first.
`to_variant_object(CAST(s AS STRUCT<a INT, b INT>))` → `{"a":1,"b":null}`;
for arrays, `CAST(array(NULL) AS ARRAY<STRING>)::VARIANT` → `[null]`.

**Deliberate or omission?** Undocumented. One hint toward omission: CHAR and
VARCHAR get an explicit `=> false` case, NullType only hits the default — and
the encoding would be trivially well-defined (every value = variant null,
type ID 0). So the JIRA stands: *support NullType in variant construction, or
document the rationale.* For the OSS repro, use `to_variant_object`/arrays,
since OSS rejects the struct CAST form entirely.

*Attachments: evidence dossier (PDF, all six runs with cluster configs and
timestamps) + raw notebook exports for DBR 16.4 LTS (Spark 3.5.2) and DBR 19
Beta (4.2.0). Identical results everywhere tested; only DBR 15.3/15.4 remain
unverified.*