- 4745 Views
- 3 replies
- 1 kudos
Resolved! Tuning `CrossValidator` spark job performance
I am running a 3-fold cross validation of an ML pipeline that utilizes `GBTClassifier` as the final step. It takes 18 hours to run and I am looking for feedback into how to improve the performance as I expect this to go faster.For context here is the...
- 4745 Views
- 3 replies
- 1 kudos
- 1 kudos
Hello @Assaad Mrad​ , So this looks like trying to decide between putting the pipeline in the cross validator or the cross validator in the pipeline. Since you are doing the polynomial expansion as part of the pipeline you might want to consider putt...
- 1 kudos
- 1613 Views
- 0 replies
- 2 kudos
2021-08-Best-Practices-for-Your-Data-Architecture-v3-OG-1200x628
Thanks to everyone who joined the Best Practices for Your Data Architecture session on Optimizing Data Performance. You can access the on-demand session recording here and the pre-run performance benchmarks using the Spark UI Simulator. Proper cluste...
- 1613 Views
- 0 replies
- 2 kudos