https://community.databricks.com/t5/data-engineering/how-can-i-scale-my-neural-network-with-spark-i-m-building-a/m-p/25264#M17553 <P>With Spark, there are a few ways you can scale your model: </P><UL><LI>Training</LI><LI>Hyperparameter tuning</LI><LI>Inference</LI></UL><P></P><P>If you're looking to train one model across multiple workers, you can leverage Horovod. It's an open source project designed to simplify distributed neural network training, and supports Keras/TF/PyTorch/MXNet. See the docs for <A href="https://docs.databricks.com/applications/machine-learning/train-model/distributed-training/horovod-runner.html" alt="https://docs.databricks.com/applications/machine-learning/train-model/distributed-training/horovod-runner.html" target="_blank">HorovodRunner</A>.</P><P></P><P>If you're looking to train many candidate models in parallel, you can use <A href="http://hyperopt.github.io/hyperopt/scaleout/spark/" alt="http://hyperopt.github.io/hyperopt/scaleout/spark/" target="_blank">HyperOpt with SparkTrials</A>. Check out this fantastic <A href="https://databricks.com/blog/2021/04/15/how-not-to-tune-your-model-with-hyperopt.html" alt="https://databricks.com/blog/2021/04/15/how-not-to-tune-your-model-with-hyperopt.html" target="_blank">blog</A> on best practices on best practices and tips on setting parallelism for SparkTrials.</P><P></P><P>You can always create a Spark UDF (super easy if you MLflow, e.g. <A href="https://mlflow.org/docs/latest/python_api/mlflow.pyfunc.html#mlflow.pyfunc.spark_udf" alt="https://mlflow.org/docs/latest/python_api/mlflow.pyfunc.html#mlflow.pyfunc.spark_udf" target="_blank">mlflow.pyfunc.spark_udf</A>) to trivially do inference in parallel for batch/streaming use cases.</P> Thu, 10 Jun 2021 18:35:04 GMT User16788317454 2021-06-10T18:35:04Z https://community.databricks.com/t5/data-engineering/how-can-i-scale-my-neural-network-with-spark-i-m-building-a/m-p/25263#M17552 Thu, 10 Jun 2021 17:59:37 GMT https://community.databricks.com/t5/data-engineering/how-can-i-scale-my-neural-network-with-spark-i-m-building-a/m-p/25263#M17552 j_weaver 2021-06-10T17:59:37Z https://community.databricks.com/t5/data-engineering/how-can-i-scale-my-neural-network-with-spark-i-m-building-a/m-p/25264#M17553 <P>With Spark, there are a few ways you can scale your model: </P><UL><LI>Training</LI><LI>Hyperparameter tuning</LI><LI>Inference</LI></UL><P></P><P>If you're looking to train one model across multiple workers, you can leverage Horovod. It's an open source project designed to simplify distributed neural network training, and supports Keras/TF/PyTorch/MXNet. See the docs for <A href="https://docs.databricks.com/applications/machine-learning/train-model/distributed-training/horovod-runner.html" alt="https://docs.databricks.com/applications/machine-learning/train-model/distributed-training/horovod-runner.html" target="_blank">HorovodRunner</A>.</P><P></P><P>If you're looking to train many candidate models in parallel, you can use <A href="http://hyperopt.github.io/hyperopt/scaleout/spark/" alt="http://hyperopt.github.io/hyperopt/scaleout/spark/" target="_blank">HyperOpt with SparkTrials</A>. Check out this fantastic <A href="https://databricks.com/blog/2021/04/15/how-not-to-tune-your-model-with-hyperopt.html" alt="https://databricks.com/blog/2021/04/15/how-not-to-tune-your-model-with-hyperopt.html" target="_blank">blog</A> on best practices on best practices and tips on setting parallelism for SparkTrials.</P><P></P><P>You can always create a Spark UDF (super easy if you MLflow, e.g. <A href="https://mlflow.org/docs/latest/python_api/mlflow.pyfunc.html#mlflow.pyfunc.spark_udf" alt="https://mlflow.org/docs/latest/python_api/mlflow.pyfunc.html#mlflow.pyfunc.spark_udf" target="_blank">mlflow.pyfunc.spark_udf</A>) to trivially do inference in parallel for batch/streaming use cases.</P> Thu, 10 Jun 2021 18:35:04 GMT https://community.databricks.com/t5/data-engineering/how-can-i-scale-my-neural-network-with-spark-i-m-building-a/m-p/25264#M17553 User16788317454 2021-06-10T18:35:04Z