1) You might need to cache the dataframe so it's not recomputing for the write

2) What type of cloud storage are you using? We've noticed slow delta writes as well. We are using Azure standard storage which is backed by spinning disks. It's limited to 60MiB per second. We been wondering if our issue is being IO bound and if switching to premium SSD storage would help. But, I'm not very familiar with SparkUI and haven't been able to find a way to prove it.  Based on your numbers 2GB at 60MB/s should only take 30 seconds. If you are getting 60Mb/s it would be 5 minutes. But, Azure Blob seems to scale fairly well across multiple files.