Topics with Label: Tuning

by mradassaad • New Contributor III

05-03-2022 9:44:08 AM

2201 Views
4 replies
1 kudos

Resolved! Tuning `CrossValidator` spark job performance

I am running a 3-fold cross validation of an ML pipeline that utilizes `GBTClassifier` as the final step. It takes 18 hours to run and I am looking for feedback into how to improve the performance as I expect this to go faster.For context here is the...

Machine Learning

2201 Views
4 replies
1 kudos

05-03-2022 9:44:08 AM

View Replies

Latest Reply

Kaniz
Community Manager

05-13-2022 3:26:29 AM

1 kudos

Hi @Assaad Mrad , Just a friendly follow-up. Do you still need help, or @Chris Chalcraft 's response help you to find the solution? Please let us know.

1 kudos

05-13-2022 3:26:29 AM

3 More Replies

by Wayne • New Contributor III

04-27-2022 9:05:44 AM

903 Views
2 replies
3 kudos

Question Submitted How to tune a job to avoid paying extra cost for EXPAND DISK? Is it due to the shuffle or data skew? Is there a way to configure the workers with larger disk? If not having EXPAND DISK, it will fail since no space left on the disk.

Machine Learning

903 Views
2 replies
3 kudos

04-27-2022 9:05:44 AM

View Replies

Latest Reply

Wayne
New Contributor III

05-05-2022 6:54:43 AM

3 kudos

No error, just seeing the EXPAND DISK in cluster event logs. This is just a regular spark application. I am not sure if the cloud storage matters - a spark application uses it as input and output.

3 kudos

05-05-2022 6:54:43 AM

1 More Replies

Databricks

Forum Posts

Resolved! Tuning `CrossValidator` spark job performance

Question Submitted How to tune a job to avoid paying extra cost for EXPAND DISK? Is it due to the shuffle or data skew? Is there a way to configure the workers with larger disk? If not having EXPAND DISK, it will fail since no space left on the disk.