Feature Store with Spark Pipeline

haseeb2001 — Fri, 16 Feb 2024 20:21:45 GMT

Hi,

I am using a spark pipeline having stages VectoreAssembler, StandardScalor, StringIndexers, VectorAssembler, GbtClassifier. And then logging this pipeline using feature store log_model function as follows:

fe = FeatureStoreClient() // I have tried this using FeatureStoreEngineeringClient too

After defining lookups and creating a training_set, I am logging this model using:

fe.log_model ( model=model_pipeline, artifact_path = "test_model", flavor = mlflow.spark, training_set = training_set, registered_model_name = "registery_name")

After logging this model, I am using fe.score function to get results on my test data. But I am getting the following error:

Re: Feature Store with Spark Pipeline

haseeb2001 — Mon, 19 Feb 2024 15:35:25 GMT

Hi @Retired_mod , thanks for your response.

The issue I am facing is during fe.score_batch. I have tried logging this pipeline using mlflow only and then tested it for inference too and it worked fine. The issue appears only when I use feature store batch scoring.

I have noticed that when I applied score it used python_function as the backend flavor, while I have registered my model using spark flavor. Any thoughts on this?

topic Feature Store with Spark Pipeline in Get Started Discussions

Feature Store with Spark Pipeline

Re: Feature Store with Spark Pipeline