Hubert-Dudek
Databricks MVP

‎10-03-2022 12:57 PM

  • It is a single file that has 16GB? Maybe it could be good to split it into some partitions with sense (partitions on the disk, so not everything needs to be read or written to).
  • If you have to write that 16GB data at once, please check that before the write is repartitioned (partitions on spark) to the number of cores (so 40) so every core process chunk of data, and then check are that 40 files similar in size (to avoid data skews)
  • Check for data spills in Spark UI as they mean writing shuffle partitions from RAM to disks and back. (25th, 50th, and 75th percentile should be similar). Increase shuffle partitions if they have to be frequently written to disk.

My blog: https://databrickster.medium.com/