Serving a custom transformer class via a pyfunc wrapper for a pyspark recommendation model

Nishat — Tue, 02 Apr 2024 07:56:41 GMT

I am trying to serve an ALS pyspark model with a custom transformer(for generating user-specific recommendations) via a pyfunc wrapper. Although I can successfully score the logged model, the serving endpoint is throwing the following error.

URI '/model/artifacts/./sparkml' does not point to the current DFS.
File '/model/artifacts/./sparkml' not found on DFS. Will attempt to upload the file.
An error occurred while loading the model. 'NoneType' object has no attribute 'jvm'.

Following is a brief summary of the code snippets used for model training, logging, and the setup for model serving:

Model training and logging to MLflow Model Registry with additional requirements and code paths specified.
Custom PySpark ML Transformer implementation for generating user-specific recommendations.
Python pyfunc wrapper for serving the model
Attempt to serve the model using Databricks' MLflow serving feature, leading to the aforementioned error

topic Serving a custom transformer class via a pyfunc wrapper for a pyspark recommendation model in Machine Learning

Serving a custom transformer class via a pyfunc wrapper for a pyspark recommendation model