https://community.databricks.com/t5/data-engineering/i-have-a-single-node-xgboost-model-written-in-python-how-can-i/m-p/25298#M17578 <P>If you are talking about distributed training of a single XGBoost model, there is no built-in capability in SparkML. SparkML supports&nbsp;<A href="https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.ml.classification.GBTClassifier.html?highlight=gbt#pyspark.ml.classification.GBTClassifier" alt="https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.ml.classification.GBTClassifier.html?highlight=gbt#pyspark.ml.classification.GBTClassifier" target="_blank">gradient boosted trees</A>, but not XGBoost specifically. However, there are 3rd party packages, such as&nbsp;<A href="https://xgboost.readthedocs.io/en/latest/jvm/xgboost4j_spark_tutorial.html" alt="https://xgboost.readthedocs.io/en/latest/jvm/xgboost4j_spark_tutorial.html" target="_blank">XGBoost4J</A>&nbsp;that you can use. Currently, there is no Python API for it, but you can access it via Scala/Java. See the Databricks&nbsp;<A href="https://docs.databricks.com/applications/machine-learning/train-model/xgboost.html" alt="https://docs.databricks.com/applications/machine-learning/train-model/xgboost.html" target="_blank">docs</A>&nbsp;for a more complete example. </P><P></P><P>If you want to scale the hyperparameter tuning, you can use HyperOpt with single node XGBoost models in Python, or you can always do distributed inference via a Spark UDF.</P> Thu, 10 Jun 2021 18:41:45 GMT j_weaver 2021-06-10T18:41:45Z https://community.databricks.com/t5/data-engineering/i-have-a-single-node-xgboost-model-written-in-python-how-can-i/m-p/25297#M17577 Thu, 10 Jun 2021 17:53:28 GMT https://community.databricks.com/t5/data-engineering/i-have-a-single-node-xgboost-model-written-in-python-how-can-i/m-p/25297#M17577 User16788317454 2021-06-10T17:53:28Z https://community.databricks.com/t5/data-engineering/i-have-a-single-node-xgboost-model-written-in-python-how-can-i/m-p/25298#M17578 <P>If you are talking about distributed training of a single XGBoost model, there is no built-in capability in SparkML. SparkML supports&nbsp;<A href="https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.ml.classification.GBTClassifier.html?highlight=gbt#pyspark.ml.classification.GBTClassifier" alt="https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.ml.classification.GBTClassifier.html?highlight=gbt#pyspark.ml.classification.GBTClassifier" target="_blank">gradient boosted trees</A>, but not XGBoost specifically. However, there are 3rd party packages, such as&nbsp;<A href="https://xgboost.readthedocs.io/en/latest/jvm/xgboost4j_spark_tutorial.html" alt="https://xgboost.readthedocs.io/en/latest/jvm/xgboost4j_spark_tutorial.html" target="_blank">XGBoost4J</A>&nbsp;that you can use. Currently, there is no Python API for it, but you can access it via Scala/Java. See the Databricks&nbsp;<A href="https://docs.databricks.com/applications/machine-learning/train-model/xgboost.html" alt="https://docs.databricks.com/applications/machine-learning/train-model/xgboost.html" target="_blank">docs</A>&nbsp;for a more complete example. </P><P></P><P>If you want to scale the hyperparameter tuning, you can use HyperOpt with single node XGBoost models in Python, or you can always do distributed inference via a Spark UDF.</P> Thu, 10 Jun 2021 18:41:45 GMT https://community.databricks.com/t5/data-engineering/i-have-a-single-node-xgboost-model-written-in-python-how-can-i/m-p/25298#M17578 j_weaver 2021-06-10T18:41:45Z