Please try to use the extract method to explode the JSON by extracting specific fields and handling optional nested fields by using the method for special scenarios  - when(col("file. F").isNull(), None).otherwise(col("file.F.ROW.F1")).alias("F1")