Hi Databricks Community,
I’m trying to deploy a model serving endpoint that uses Databricks Feature Store (Unity Catalog, online tables).
- My offline and online feature tables are created and visible in Databricks.
- The model is logged with FeatureEngineeringClient.log_model and feature lookups added.
- Training and batch inference works fine (these are using the offline feature tables).
- When I deploy the model serving endpoint, it fails with the following error:
"Exception: Internal error: _USER not found for feature table catalog.schema.nyc_taxi_pickup_features_spn."
Error logs from Databricks Serving UI (redacted):
ENDPOINT_UPDATE_EVENT
Endpoint update failed for endpoint `featurestores-endpoint-spn`, config version 1.
SERVED_ENTITY_SERVICE_EVENT
featurestores-model-12
Served entity creation failed for served entity `featurestores-model-12`, config version 1. Error message: Model server failed to load the model. Please see service logs for more information.
SERVED_ENTITY_SERVICE_EVENT
Served entity `featurestores-model-12` entered DEPLOYMENT_FAILED state: Failed to load the model. Exit code 1.
SERVED_ENTITY_SERVICE_EVENT
Served entity `featurestores-model-12` entered DEPLOYMENT_CREATING state: Deploying
SERVED_ENTITY_SERVICE_EVENT
Served entity `featurestores-model-12` entered DEPLOYMENT_CREATING state: Provisioning resources
SERVED_ENTITY_CREATION_EVENT
featurestores-model-12
Served entity created for served entity `featurestores-model-12`, config version 1.
ENDPOINT_UPDATE_EVENT
featurestores-model-12
System role `***********************` created successfully for Online Feature Store `mlops-feature-store-instance-spn`.
ENDPOINT_UPDATE_EVENT
featurestores-model-12
System service principal creation with ID `***************************` succeeded for served entity `featurestores-model-12`, config version 1.
SERVED_ENTITY_CREATION_EVENT
featurestores-model-12
Linked with Databricks Online Store table: catalog.schema.nyc_taxi_dropoff_features_spn_online
SERVED_ENTITY_CREATION_EVENT
featurestores-model-12
Linked with Databricks Online Store table: catalog.schema.int.nyc_taxi_pickup_features_spn_online
Full stack trace excerpt:
File ".../databricks/feature_store/mlflow_model.py", line 161, in __init__
self.ft_to_lookup_client = self._create_lookup_clients(self.ft_metadata)
File ".../databricks/feature_store/online_lookup_client.py", line 207, in _generate_lookup_engine
creds = load_credentials_from_env(first_online_feature_table)
File ".../databricks/feature_store/online_lookup_client.py", line 102, in get_env_var
raise Exception(
Exception: Internal error: _USER not found for feature table catalog.schema.nyc_taxi_pickup_features_spn.
Environment:
- databricks-sdk==0.62
- databricks-feature-engineering==0.13
- mlflow==3.0.1
How I logged the model:
- In my training notebook I used fe.log_model with the correct FeatureLookup to log the model.
- The model is registered in Unity Catalog and the feature metadata appears correct.
- I create the online tables in a notebook using FeatureEngineeringClient.publish_table.
- I deploy the endpoint using the Python SDK (WorkspaceClient).
- I do not know where to get the required credentials (user, password, host, etc.) for the online feature store, nor how to set them as environment variables for the endpoint. Do I need to do this manually?
Questions:
How is it possible to use the Online feature tables in a model trained and logged with the FeatureEngineeringClient?
What is this _USER suffix parameter that is required and why was it not packaged properly be the FeatureEngineeringClient? I'm running the processes from pipeline so the service principal is the creator and thus the owner of both the model, offline and online feature tables, why is the automatic lookup not working?
Any help or guidance would be greatly appreciated!
