Issue with MongoDB Void Null Type in Databricks

[shan-databricks]

Facing an issue with the MongoDB Void/Null type in Databricks, which requires explicit casting or conversion to an array or struct of strings. Looking for guidance on how to handle this data type when reading from MongoDB and writing to a Delta table, as Delta does not support the Void/Null type.

[lingareddy_Alva]

Hi @shan-databricks 

When MongoDB has fields where all sampled documents are null, Spark cannot infer a data type, so it assigns NullType (void). Delta Lake rejects this because it needs a concrete type for Parquet storage.

My suggestion would be the Fix — Two Layers

Layer 1 — Explicit Schema on Read
Declare the schema yourself before reading from MongoDB. Spark skips inference entirely and uses your declared types
even if values are null at runtime, the column type is concrete and Delta accepts it.

Layer 2 — Defensive Null Cast (Safety Net)
After reading, scan all columns and cast any surviving NullType to StringType.
This catches edge cases where the MongoDB connector overrides your declared schema internally for certain field patterns.

[balajij8]

You can handle VOID columns directly if you are on Databricks Runtime 18.2 or later for batch writes.

More details here

You can explicitly cast or replace VOID/NULL columns with appropriate types when reading from MongoDB

df = df.withColumn("files_col", 
                    when(col("files_col").isNull(), lit(None).cast(StringType()))
                    .otherwise(col("files_col")))

VOID columns are not supported in streaming writes.

You can identify the MongoDB fields containing null/void values & create a mapping of column names to target Delta types (String etc). Then apply explicit casting using the code above & enable mergeSchema when writing to Delta for schema evolution. You can also use coalesce with default values for critical fields.

Always casting VOID columns to explicit types before writing to Delta regardless of runtime to ensure compatibility and avoid potential issues.