cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 

Are UDFs necessary for applying models from ML libraries at scale ?

anvil
New Contributor II

Hello,

I recently finished the "scalable machine learning with apache spark" course and saw that SKLearn models could be applied faster in a distributed manner when used in pandas UDFs or with mapInPandas() method.

Spark MLlib models don't need this kind of refactoring since they are made for distributed executions but I was wondering if this kind of UDF was necessary for other libraries such as TensorFlow, PyTorch, SpaCy, Keras, etc.

Thank you !

3 REPLIES 3

Hubert-Dudek
Esteemed Contributor III

MlLib is in the maintenance model. Currently, Spark ML is used mainly. Creating the model is not using UDFs in most cases https://www.databricks.com/spark/getting-started-with-apache-spark/machine-learning, but anyway, UDF is usually run in a distributed way. For example, when you append data to your table, you can use UDF to run prediction using a registered model (even on a real-time stream):

import mlflow
predict = mlflow.pyfunc.spark_udf(spark, model_uri=f"runs:/{run_id}/model")
predDF = testDF.withColumn("prediction", predict(*testDF.columns))

Devarsh
Contributor

UDFs are not necessarily required for applying models from ML libraries at scale, but they can provide some benefits in terms of performance and ease of use.

When using other libraries such as TensorFlow, PyTorch, SpaCy, Keras, etc., they are not optimized for distributed processing by default. In this case, using UDFs or the mapInPandas() method can provide a way to scale the models efficiently, by parallelizing the processing across the Spark cluster.

Anyways, it ultimately depends on the specific requirements of your project.

Manoj12421
Valued Contributor II

MlLib is in the maintenance model and udf is not used by creating model in most cases

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!