- 1631 Views
- 1 replies
- 0 kudos
For tuning hyperparameters with Apache Spark ML / MLlib, when should I use Spark ML's built-in tuning algorithms vs. Hyperopt?
When should I use Spark ML's CrossValidator or TrainValidationSplit, vs. a separate tuning tool such as Hyperopt?
- 1631 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Both are valid choices. By default, I'd recommend using Hyperopt nowadays. Here's the rationale, as pros & cons of each.Spark ML's built-in toolsPros: These fit the Spark ML Pipeline framework, so you can keep using the same type of APIs.Cons: Thes...
- 0 kudos
- 2314 Views
- 1 replies
- 0 kudos
- 2314 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
These terms are borrowed from scikit-learn, and the idea is the same. A transformer is just a component of a pipeline that transforms the data in some way. An estimator is also a transfomer, but one that additionally needs to be 'fit' on data before ...
- 0 kudos