NULL rows getting inserted in delta table- Schema ...

AanchalSoni · ‎03-23-2026

I'm trying to add _metadata column while reading a json file:

from pyspark.sql.functions import col

from pyspark.sql.types import StructType, StructField, LongType, TimestampType

df_accounts_read = spark.readStream.format("cloudFiles").\

option("cloudFiles.format", "json").\

option("multiline", "true").\

option("schemaLocation", "/Volumes/capstone_project/raw/raw_data/Accounts/Schema/").\

schema(schema).\

option("cloudFiles.rescuedDataColumn", "_rescued_data").\

option("mode", "dropMalformed").\

load("/Volumes/capstone_project/raw/raw_data/Accounts").\

withColumn("_metadata", col("_metadata").cast(

StructType(

[

StructField("file_path", StringType()),

StructField("file_name", StringType()),

StructField("file_size", LongType()),

StructField("file_block_start", LongType()),

StructField("file_block_length", LongType()),

StructField("file_modification_time", TimestampType())

]

)))

The schema has been explicitly defined for the json file. I'm writing the reads to a delta table using auto loader. When I check the table for null records, I see that metadata column is creating 3 null rows. Please refer screenshot. I'm unable to merge schema of metadata column. Please guide, how can I avoid these NULL rows?

Write stream code:

df_accounts_read.writeStream.format("delta").\

option("checkpointLocation", "/Volumes/capstone_project/raw/raw_data/Accounts/Checkpoint/").\

option("cloudFiles.schemaEvolutionMode", "addNewColumns").\

option("mergeSchema", "true").\

trigger(once=True).\

toTable("capstone_project.bronze.b_accounts")

NULL rows getting inserted in delta table- Schema mismatch