Are UDFs necessary for applying models from ML libraries at scale ?

[anvil]

Hello,

I recently finished the "scalable machine learning with apache spark" course and saw that SKLearn models could be applied faster in a distributed manner when used in pandas UDFs or with mapInPandas() method.

Spark MLlib models don't need this kind of refactoring since they are made for distributed executions but I was wondering if this kind of UDF was necessary for other libraries such as TensorFlow, PyTorch, SpaCy, Keras, etc.

Thank you !

[Hubert-Dudek]

MlLib is in the maintenance model. Currently, Spark ML is used mainly. Creating the model is not using UDFs in most cases https://www.databricks.com/spark/getting-started-with-apache-spark/machine-learning, but anyway, UDF is usually run in a distributed way. For example, when you append data to your table, you can use UDF to run prediction using a registered model (even on a real-time stream):

import mlflow
predict = mlflow.pyfunc.spark_udf(spark, model_uri=f"runs:/{run_id}/model")
predDF = testDF.withColumn("prediction", predict(*testDF.columns))

[Devarsh]

UDFs are not necessarily required for applying models from ML libraries at scale, but they can provide some benefits in terms of performance and ease of use.

When using other libraries such as TensorFlow, PyTorch, SpaCy, Keras, etc., they are not optimized for distributed processing by default. In this case, using UDFs or the mapInPandas() method can provide a way to scale the models efficiently, by parallelizing the processing across the Spark cluster.

Anyways, it ultimately depends on the specific requirements of your project.

[Manoj12421]

MlLib is in the maintenance model and udf is not used by creating model in most cases