I was able to save it as a delta file. You need to specify the minReaderVersion and minWriterVersion as well. E.g.spark_df.write.format("delta").mode('overwrite').option("delta.columnMapping.mode", "name").option('delta.minReaderVersion', '2').option...