I think problem is in overwrite as when you overwrite it overwrites all folders. Solution is to mix append with dynamic overwrite so it will overwrite only folders which have data and doesn't affect old partitions:

spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")