cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Is Feature Store packaged model compatible with Spark UDF?

Chengcheng
New Contributor III

Hi, I tried to deploy a Feature Store packaged model into Delta Live Table using mlflow.pyfunc.spark_udf in Azure Databricks.

This model is built by Databricks autoML with joined Feature Table inside it.

And I'm trying to make prediction using the following code in my notebook. (and also DLT)

# Load model as a Spark UDF. Override result_type if the model does not return double values.
loaded_model = mlflow.pyfunc.spark_udf(spark, model_uri=logged_model, result_type='double', env_manager='virtualenv')
 
# Predict on a Spark DataFrame.
df.withColumn('predictions', loaded_model(struct(*map(col, df.columns))))

But I get the following error:

Exception: Internal error: Online feature table information could not be found for feature tables.

This feature table does exist and inference using

predictions_df = fs.score_batch(logged_model,  batch_input_df, result_type="double")

fs.score_batch works. With a warning about Spark UDF as following:

WARNING mlflow.pyfunc: Calling `spark_udf()` with `env_manager="local"` does not recreate the same environment that was used during training, which may lead to errors or inaccurate predictions. We recommend specifying `env_manager="conda"`, which automatically recreates the environment that was used to train the model and performs inference in the recreated environment.

I also tried another Feature Store packaged model which packed by myself but it comes with the same error.

So my question is:

Is the spark_udf function compatible with Feature Store packaged model at this moment?

If it does, how should I fix this error and deploy it to DLT?

1 REPLY 1

Anonymous
Not applicable

Hi @Chengcheng Guo​ 

Great to meet you, and thanks for your question!

Let's see if your peers in the community have an answer to your question. Thanks.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!