Pallav
New Contributor II

Most of the optimisations can be done while selecting the number of partitions we can to create for data, too many would cause a large shuffle operation on wide dependency operations and too less would cause less parallelisation. To minimise the time taken during shuffle operations , use zordering so that data having high chances of falling under the same aggregation, are located on the same or nearby partitions.