cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 

Feature Store Model Serving endpoint

NaeemS
New Contributor III

Hi,

I am trying to deploy my model which was logged by featureStoreEngineering client as a serving endpoint in Databricks. But I am facing following error:

 

 

 The Databricks Lookup client from databricks-feature-lookup and Databricks Feature Store client from databricks-feature-engineering cannot be installed in the same python environment. 

 

 

 

When I log my model using FeatureStoreEngineeringClient databricks-feature-lookup is added by default in my requirements file, and when I load that model in other environment I get this error. Also, when I add databricks-feature-engineering ad dependency I get the same error in my model serving endpoint creation. Also, if I don't add databricks-feature-engineering as a dependency I will get other errors while creation of my serving endpoint. 

 

model_impl = importlib.import_module(conf[MAIN])._load_pyfunc(data_path) [cc8d72wmj6] File "/opt/conda/envs/mlflow-env/lib/python3.10/site-packages/mlflow/spark/__init__.py", line 898, in _load_pyfunc [cc8d72wmj6] spark = _create_local_spark_session_for_loading_spark_model() [cc8d72wmj6] File "/opt/conda/envs/mlflow-env/lib/python3.10/site-packages/mlflow/utils/_spark_utils.py", line 131, in _create_local_spark_session_for_loading_spark_model [cc8d72wmj6] .getOrCreate() [cc8d72wmj6] File "/opt/conda/envs/mlflow-env/lib/python3.10/site-packages/pyspark/sql/session.py", line 497, in getOrCreate [cc8d72wmj6] sc = SparkContext.getOrCreate(sparkConf) [cc8d72wmj6] File "/opt/conda/envs/mlflow-env/lib/python3.10/site-packages/pyspark/context.py", line 515, in getOrCreate [cc8d72wmj6] SparkContext(conf=conf or SparkConf()) [cc8d72wmj6] File "/opt/conda/envs/mlflow-env/lib/python3.10/site-packages/pyspark/context.py", line 201, in __init__ [cc8d72wmj6] SparkContext._ensure_initialized(self, gateway=gateway, conf=conf) [cc8d72wmj6] File "/opt/conda/envs/mlflow-env/lib/python3.10/site-packages/pyspark/context.py", line 436, in _ensure_initialized [cc8d72wmj6] SparkContext._gateway = gateway or launch_gateway(conf) [cc8d72wmj6] File "/opt/conda/envs/mlflow-env/lib/python3.10/site-packages/pyspark/java_gateway.py", line 107, in launch_gateway [cc8d72wmj6] raise PySparkRuntimeError( [cc8d72wmj6] pyspark.errors.exceptions.base.PySparkRuntimeError: [JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port number. [cc8d72wmj6] [2024-06-20 21:04:06 +0000] [10] [ERROR] Error handling request /v2/health/ready I would appreciate any help in solving this issue.

 

 

Thanks,

6 REPLIES 6

Kaniz_Fatma
Community Manager
Community Manager

Hi @NaeemS, It seems you’re encountering an issue related to conflicting dependencies when deploying your model as a serving endpoint in Databricks.

Specifically, the Databricks Lookup client from databricks-feature-lookup and the Databricks feature store client from databricks-feature-engineering cannot be installed in the same Python environment.

I hope this helps you troubleshoot the issue! If you need further assistance, feel free to ask. 

NaeemS
New Contributor III

Hi @Kaniz_Fatma ,

Thanks for your response. But I have limited access to shared cluster and can not use it for this purpose. Can you tell me how I can prevent databricks-feature-lookup from being added in my requirements file when I am logging my model using FeatureStoreEngineeringClient which is causing this issue. 

Or any other workaround for solving this issue with single user cluster.

 

robbe
New Contributor III

Hi @NaeemS , but the issue should not arise from having databricks-feature-lookup in your serving endpoint, quite the opposite. You should have the databricks-feature-lookup dependency but not the databricks-feature-engineering dependency.

Can you please elaborate more on the error that you mentioned in the OP, perhaps including some reproducible code?

NaeemS
New Contributor III

Hi @robbe , 
I am facing following errors when creating serving endpoint for my spark pipeline 

5548dsptvc] raise self._exception
[5548dsptvc] File "/opt/conda/envs/mlflow-env/lib/python3.10/site-packages/mlflowserving/scoring_server/__init__.py", line 194, in get_model_option_or_exit
[5548dsptvc] self.model = self.model_future.result()
[5548dsptvc] File "/opt/conda/envs/mlflow-env/lib/python3.10/concurrent/futures/_base.py", line 451, in result
[5548dsptvc] return self.__get_result()
[5548dsptvc] File "/opt/conda/envs/mlflow-env/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
[5548dsptvc] raise self._exception
[5548dsptvc] File "/opt/conda/envs/mlflow-env/lib/python3.10/site-packages/mlflowserving/scoring_server/__init__.py", line 194, in get_model_option_or_exit
[5548dsptvc] self.model = self.model_future.result()
[5548dsptvc] File "/opt/conda/envs/mlflow-env/lib/python3.10/concurrent/futures/_base.py", line 451, in result
[5548dsptvc] return self.__get_result()
[5548dsptvc] File "/opt/conda/envs/mlflow-env/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
[5548dsptvc] raise self._exception
[5548dsptvc] File "/opt/conda/envs/mlflow-env/lib/python3.10/site-packages/mlflowserving/scoring_server/__init__.py", line 194, in get_model_option_or_exit
[5548dsptvc] self.model = self.model_future.result()
[5548dsptvc] File "/opt/conda/envs/mlflow-env/lib/python3.10/concurrent/futures/_base.py", line 451, in result
[5548dsptvc] return self.__get_result()

I have tested creating a sklearn model in my environment which is working fine, but for the spark pipeline I am unable to do so , following is my requirement file which is being logged with my model.

mlflow==2.14.1
numpy==1.21.5
pyspark==3.5.1
scipy==1.9.1
databricks-feature-lookup==0.*


databricks-feature-lookup 1.2.14 is being installed in my environment as the earlier versions has been deprecated.

Thanks,

robbe
New Contributor III

Hi @NaeemS, it's hard to say given how uninformative the error is. I will try to give it a go next week but maybe you can help me by answering a few questions:

  • Can you paste the exact line of code that trigger the error?
  • Does the sklearn model also use the Feature Store? Is the only difference in the library used (pyspark vs sklearn)?
  • Is your Online Feature Store correctly configured with the right credentials to retrieve the latest feature set?
  • What happens if you remove the
    databricks-feature-lookup==0.*​
    from your requirements file?

NaeemS
New Contributor III

Hi @robbe , 

My model was logged successfully using feature engineering client. The issue appears while creating a serving endpoint for that model.
For the Sklearn model I used a simple Random Forest model and for the spark model I am logging a pipeline with multiple steps like imputer, indexer, assembler and model.
Also, I am not using 3rd part feature stores here. I am using online tables within Unity Catalog, this is relatively a new feature introduced by Databricks. I am using same store with sklearn model and it is working fine, the issue appears only with spark pipeline.

While logging my model I am not specifying databricks-feature-lookup as dependency, it is being added by default with my model even if I give my own requirements file while logging the model.

Thanks

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!