I was able to save it as a delta file. You need to specify the minReaderVersion and minWriterVersion as well. E.g.

spark_df.write.format("delta").mode('overwrite').option("delta.columnMapping.mode", "name").option('delta.minReaderVersion', '2').option('delta.minWriterVersion', '5').save('/path/to/table')

Reference: Table protocol versioning — Delta Lake Documentation