Question Submitted How to tune a job to avoid paying extra cost for EXPAND DISK? Is it due to the shuffle or data skew? Is there a way to configure the workers with larger disk? If not having EXPAND DISK, it will fail since no space left on the disk.

[Anonymous]

Your worker disk size should almost never matter. You should be using cloud storage such as S3/ADL2/GCS. What operations are you running and what error message are you getting?

[Wayne]

No error, just seeing the EXPAND DISK in cluster event logs. This is just a regular spark application. I am not sure if the cloud storage matters - a spark application uses it as input and output.