cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 

Serving a custom transformer class via a pyfunc wrapper for a pyspark recommendation model

Nishat
New Contributor

I am trying to serve an ALS pyspark model with a custom transformer(for generating user-specific recommendations) via a pyfunc wrapper. Although I can successfully score the logged model, the serving endpoint is throwing the following error.

URI '/model/artifacts/./sparkml' does not point to the current DFS.
File '/model/artifacts/./sparkml' not found on DFS. Will attempt to upload the file.
An error occurred while loading the model. 'NoneType' object has no attribute 'jvm'.

Following is a brief summary of the code snippets used for model training, logging, and the setup for model serving:

  • Model training and logging to MLflow Model Registry with additional requirements and code paths specified.
  • Custom PySpark ML Transformer implementation for generating user-specific recommendations.
  • Python pyfunc wrapper for serving the model
  • Attempt to serve the model using Databricks' MLflow serving feature, leading to the aforementioned error
0 REPLIES 0

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group