Feature Store with Spark Pipeline

haseeb2001
New Contributor II

Hi,

I am using a spark pipeline having stages VectoreAssembler, StandardScalor, StringIndexers, VectorAssembler, GbtClassifier. And then logging this pipeline using feature store log_model function as follows:

fe = FeatureStoreClient() // I have tried this using FeatureStoreEngineeringClient too

After defining lookups and creating a training_set, I am logging this model using:

 

 

fe.log_model ( model=model_pipeline, artifact_path = "test_model", flavor = mlflow.spark, training_set = training_set, registered_model_name = "registery_name")

 

 

After logging this model, I am using fe.score function to get results on my test data. But I am getting the following error:

 

image.png