Hi @saurabh18cs ,
Thank you for your reply.
I did try this.
It inserts only 1 row with nulls, however,doesn't load all 2 records.Only 1st row gets inserted.
My json file is anyway not a multiline format.
Code :
df = spark.readStream.\
format("cloudFiles")\
.option("cloudFiles.format", "json")\
.option("multiLine", "true")\
.option("cloudFiles.schemaLocation", "/Volumes/workspace/default/sys/schema")\
.schema(schema)\
.load('/Volumes/workspace/dev/input/')\
.writeStream\
.format("delta")\
.option("checkpointLocation", "/Volumes/workspace/default/sys/checkpoint")\
.option("mergeSchema", "true")\
.trigger(availableNow=True)\
.toTable("workspace.default.json_null_ml")
Output :
