Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-01-2026 06:51 AM - edited 04-01-2026 06:54 AM
Hi,
I am exploring Schema inference and Schema evolution using Autoloader.I am reading a single line json file and writing in a delta table which does not exist already (creating it on the fly), using pyspark (below is the code).
Code :
spark.readStream.\
format("cloudFiles")\
.option("cloudFiles.format", "json")\
.option("cloudFiles.schemaLocation", "/Volumes/workspace/default/sys/schema1")\
.load('/Volumes/workspace/dev/input/')\
.writeStream\
.format("delta")\
.option("mergeSchema", "true")\
.option("checkpointLocation", "/Volumes/workspace/default/sys/checkpoint1")\
.trigger(once=True)\
.toTable("workspace.dev.infer_json_new1")
Input File : user.json contains only 2 records as mentioned below WITHOUT having blank lines.Attached is the screenshot.
{"Name":"Alfred","Gender":"M","Age":14}
{"Name":"John","Gender":"M","Age":12}
{"Name":"John","Gender":"M","Age":12}
Target table :
Screenshot is attached.
Problem : Upon running the code very first time,It loads 33 null records along with actual data rows.So total 35 rows get inserted.However, after re running the code with new file,it does not inserts any null rows.
I dont understand WHY 33 null rows at first run only.
