Experiences with CatBoost Spark Integration in Production on Databricks?

Machine Learning

Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.

Hi Community,

I am currently evaluating various gradient boosting options on Databricks using production-level data, including the CatBoost Spark integration (ai.catboost:catboost-spark).

I would love to hear from others who have successfully used this specific integration for production workloads. How have you found its stability and resource requirements, particularly concerning the driver, compared to alternatives like XGBoost Spark or LightGBM (via SynapseML)?

Are there any other preferred libraries or approaches for robust gradient-boosting training within the Databricks environment?

Thank you for sharing your insights!