cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

How can I scale my neural network with spark? I'm building a fully connected tensorflow.keras model.

j_weaver
New Contributor III
 
1 ACCEPTED SOLUTION

Accepted Solutions

User16788317454
New Contributor III

With Spark, there are a few ways you can scale your model:

  • Training
  • Hyperparameter tuning
  • Inference

If you're looking to train one model across multiple workers, you can leverage Horovod. It's an open source project designed to simplify distributed neural network training, and supports Keras/TF/PyTorch/MXNet. See the docs for HorovodRunner.

If you're looking to train many candidate models in parallel, you can use HyperOpt with SparkTrials. Check out this fantastic blog on best practices on best practices and tips on setting parallelism for SparkTrials.

You can always create a Spark UDF (super easy if you MLflow, e.g. mlflow.pyfunc.spark_udf) to trivially do inference in parallel for batch/streaming use cases.

View solution in original post

1 REPLY 1

User16788317454
New Contributor III

With Spark, there are a few ways you can scale your model:

  • Training
  • Hyperparameter tuning
  • Inference

If you're looking to train one model across multiple workers, you can leverage Horovod. It's an open source project designed to simplify distributed neural network training, and supports Keras/TF/PyTorch/MXNet. See the docs for HorovodRunner.

If you're looking to train many candidate models in parallel, you can use HyperOpt with SparkTrials. Check out this fantastic blog on best practices on best practices and tips on setting parallelism for SparkTrials.

You can always create a Spark UDF (super easy if you MLflow, e.g. mlflow.pyfunc.spark_udf) to trivially do inference in parallel for batch/streaming use cases.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group