There are few options I tried out which had given me a better performance.

1. Caching the intermediate or final results so that while writing the dataframe computation does not repeat again.
2. Coalesce the results into the partitions 1x or 0.5x your number of cores and also ensure that your partitions are equal to or more than 128 MB blocks so that the writes are happening efficiently.