topic Re: Question Submitted How to tune a job to avoid paying extra cost for EXPAND DISK? Is it due to the shuffle or data skew? Is there a way to configure the workers with larger disk? If not having EXPAND DISK, it will fail since no space left on the disk. in Machine Learning https://community.databricks.com/t5/machine-learning/question-submitted-how-to-tune-a-job-to-avoid-paying-extra-cost/m-p/22009#M1243 <P>No error, just seeing the EXPAND DISK in cluster event logs. This is just a regular spark application. I am not sure if the cloud storage matters - a spark application uses it as input and output. </P> Thu, 05 May 2022 13:54:43 GMT Wayne 2022-05-05T13:54:43Z Question Submitted How to tune a job to avoid paying extra cost for EXPAND DISK? Is it due to the shuffle or data skew? Is there a way to configure the workers with larger disk? If not having EXPAND DISK, it will fail since no space left on the disk. https://community.databricks.com/t5/machine-learning/question-submitted-how-to-tune-a-job-to-avoid-paying-extra-cost/m-p/22007#M1241 Wed, 27 Apr 2022 16:05:44 GMT https://community.databricks.com/t5/machine-learning/question-submitted-how-to-tune-a-job-to-avoid-paying-extra-cost/m-p/22007#M1241 Wayne 2022-04-27T16:05:44Z Re: Question Submitted How to tune a job to avoid paying extra cost for EXPAND DISK? Is it due to the shuffle or data skew? Is there a way to configure the workers with larger disk? If not having EXPAND DISK, it will fail since no space left on the disk. https://community.databricks.com/t5/machine-learning/question-submitted-how-to-tune-a-job-to-avoid-paying-extra-cost/m-p/22008#M1242 <P>Your worker disk size should almost never matter. You should be using cloud storage such as S3/ADL2/GCS. What operations are you running and what error message are you getting?</P> Wed, 27 Apr 2022 17:23:18 GMT https://community.databricks.com/t5/machine-learning/question-submitted-how-to-tune-a-job-to-avoid-paying-extra-cost/m-p/22008#M1242 Anonymous 2022-04-27T17:23:18Z Re: Question Submitted How to tune a job to avoid paying extra cost for EXPAND DISK? Is it due to the shuffle or data skew? Is there a way to configure the workers with larger disk? If not having EXPAND DISK, it will fail since no space left on the disk. https://community.databricks.com/t5/machine-learning/question-submitted-how-to-tune-a-job-to-avoid-paying-extra-cost/m-p/22009#M1243 <P>No error, just seeing the EXPAND DISK in cluster event logs. This is just a regular spark application. I am not sure if the cloud storage matters - a spark application uses it as input and output. </P> Thu, 05 May 2022 13:54:43 GMT https://community.databricks.com/t5/machine-learning/question-submitted-how-to-tune-a-job-to-avoid-paying-extra-cost/m-p/22009#M1243 Wayne 2022-05-05T13:54:43Z