Is Feature Store packaged model compatible with Spark UDF?

Chengcheng — Thu, 15 Jun 2023 07:21:56 GMT

Hi, I tried to deploy a Feature Store packaged model into Delta Live Table using mlflow.pyfunc.spark_udf in Azure Databricks.

This model is built by Databricks autoML with joined Feature Table inside it.

And I'm trying to make prediction using the following code in my notebook. (and also DLT)

# Load model as a Spark UDF. Override result_type if the model does not return double values.
loaded_model = mlflow.pyfunc.spark_udf(spark, model_uri=logged_model, result_type='double', env_manager='virtualenv')
 
# Predict on a Spark DataFrame.
df.withColumn('predictions', loaded_model(struct(*map(col, df.columns))))

But I get the following error:

Exception: Internal error: Online feature table information could not be found for feature tables.

This feature table does exist and inference using

predictions_df = fs.score_batch(logged_model,  batch_input_df, result_type="double")

fs.score_batch works. With a warning about Spark UDF as following:

WARNING mlflow.pyfunc: Calling `spark_udf()` with `env_manager="local"` does not recreate the same environment that was used during training, which may lead to errors or inaccurate predictions. We recommend specifying `env_manager="conda"`, which automatically recreates the environment that was used to train the model and performs inference in the recreated environment.

I also tried another Feature Store packaged model which packed by myself but it comes with the same error.

So my question is:

Is the spark_udf function compatible with Feature Store packaged model at this moment?

If it does, how should I fix this error and deploy it to DLT?

Re: Is Feature Store packaged model compatible with Spark UDF?

Anonymous — Sat, 17 Jun 2023 09:34:31 GMT

Hi @Chengcheng Guo

Great to meet you, and thanks for your question!

Let's see if your peers in the community have an answer to your question. Thanks.

topic Re: Is Feature Store packaged model compatible with Spark UDF? in Data Engineering

Is Feature Store packaged model compatible with Spark UDF?

Re: Is Feature Store packaged model compatible with Spark UDF?