Databricks Model Serving Endpoint Fails: “_USER not found for feature table”

[peternagy]

Hi Databricks Community,

I’m trying to deploy a model serving endpoint that uses Databricks Feature Store (Unity Catalog, online tables).

• My offline and online feature tables are created and visible in Databricks.
• The model is logged with FeatureEngineeringClient.log_model and feature lookups added.
• Training and batch inference works fine (these are using the offline feature tables).
• When I deploy the model serving endpoint, it fails with the following error:

"Exception: Internal error: _USER not found for feature table catalog.schema.nyc_taxi_pickup_features_spn."

Error logs from Databricks Serving UI (redacted):

ENDPOINT_UPDATE_EVENT

Endpoint update failed for endpoint `featurestores-endpoint-spn`, config version 1.

SERVED_ENTITY_SERVICE_EVENT

featurestores-model-12

Served entity creation failed for served entity `featurestores-model-12`, config version 1. Error message: Model server failed to load the model. Please see service logs for more information.

SERVED_ENTITY_SERVICE_EVENT

Served entity `featurestores-model-12` entered DEPLOYMENT_FAILED state: Failed to load the model. Exit code 1.

SERVED_ENTITY_SERVICE_EVENT

Served entity `featurestores-model-12` entered DEPLOYMENT_CREATING state: Deploying

SERVED_ENTITY_SERVICE_EVENT

Served entity `featurestores-model-12` entered DEPLOYMENT_CREATING state: Provisioning resources

SERVED_ENTITY_CREATION_EVENT

featurestores-model-12

Served entity created for served entity `featurestores-model-12`, config version 1.

ENDPOINT_UPDATE_EVENT

featurestores-model-12

System role `***********************` created successfully for Online Feature Store `mlops-feature-store-instance-spn`.

ENDPOINT_UPDATE_EVENT

featurestores-model-12

System service principal creation with ID `***************************` succeeded for served entity `featurestores-model-12`, config version 1.

SERVED_ENTITY_CREATION_EVENT

featurestores-model-12

Linked with Databricks Online Store table: catalog.schema.nyc_taxi_dropoff_features_spn_online

SERVED_ENTITY_CREATION_EVENT

featurestores-model-12

Linked with Databricks Online Store table: catalog.schema.int.nyc_taxi_pickup_features_spn_online

Full stack trace excerpt:

File ".../databricks/feature_store/mlflow_model.py", line 161, in __init__
self.ft_to_lookup_client = self._create_lookup_clients(self.ft_metadata)
File ".../databricks/feature_store/online_lookup_client.py", line 207, in _generate_lookup_engine
creds = load_credentials_from_env(first_online_feature_table)
File ".../databricks/feature_store/online_lookup_client.py", line 102, in get_env_var
raise Exception(
Exception: Internal error: _USER not found for feature table catalog.schema.nyc_taxi_pickup_features_spn.

Environment:

• databricks-sdk==0.62
• databricks-feature-engineering==0.13
• mlflow==3.0.1

How I logged the model:

• In my training notebook I used fe.log_model with the correct FeatureLookup to log the model.
• The model is registered in Unity Catalog and the feature metadata appears correct.

What I’ve tried:

• I create the online tables in a notebook using FeatureEngineeringClient.publish_table.
• I deploy the endpoint using the Python SDK (WorkspaceClient).
• I do not know where to get the required credentials (user, password, host, etc.) for the online feature store, nor how to set them as environment variables for the endpoint. Do I need to do this manually?

Questions:
How is it possible to use the Online feature tables in a model trained and logged with the FeatureEngineeringClient?
What is this _USER suffix parameter that is required and why was it not packaged properly be the FeatureEngineeringClient? I'm running the processes from pipeline so the service principal is the creator and thus the owner of both the model, offline and online feature tables, why is the automatic lookup not working?

Any help or guidance would be greatly appreciated!

[AbhaySingh]

Root cause in plain English

• The lookup client is trying to read SQL-style credentials like PREFIX_USER/PREFIX_PASSWORD for a third‑party online store and the “prefix” is empty, so it searches for “_USER” and fails. That auth scheme applies only to third‑party stores; Databricks Online Feature Store (Lakebase) does not require USER/PASSWORD/HOST env vars for serving.

Resolve in 6 steps

1) Use the Databricks Online Feature Store path end‑to‑end

• Make sure your feature tables are published to a Databricks Online Feature Store (Lakebase) via FeatureEngineeringClient.publish_table and that the online tables are AVAILABLE in UC. Don’t pass SQL OnlineStoreSpec or secret prefixes; just use DatabricksOnlineStore and let Databricks manage auth.

2) Confirm platform and client versions

• Use DBR 16.4 LTS ML (or Serverless) for notebooks/jobs that publish tables and log the model, and install databricks-feature-engineering >= 0.13.0 in those workflows. This is the required stack for Online Feature Stores and automatic lookup.
• If your model was logged on older clients/runtimes, re‑log it with the current databricks-feature-engineering so the serving image uses the modern lookup client (databricks-feature-lookup) and understands Lakebase metadata correctly.

3) Re‑publish feature tables correctly (if in doubt, redo)

• Before publishing: ensure primary key exists, PK columns are NOT NULL, and CDF is enabled on the offline table.
• Publish with DatabricksOnlineStore (no secrets/host/user/password). Wait for the online table status to be AVAILABLE in UC.

4) Re‑log and redeploy the model for automatic lookup

• Log via FeatureEngineeringClient.log_model with your FeatureLookup definitions. Models logged this way automatically track lineage and perform feature lookups from online stores at inference; no extra credentials are needed for Databricks Online Feature Store.

5) Remove third‑party store credentials and env vars

• On the serving endpoint, remove environment variables intended for SQL stores (for example, any PREFIX_USER/PREFIX_PASSWORD, LAKEBASE_USER/LAKEBASE_PASSWORD). These are documented for third‑party stores and will misroute the lookup client; Lakebase uses OAuth transparently and does not need them.

6) Validate endpoint permissions for the auto service principal

• Model Serving creates its own system service principal to query your online tables. Ensure it has USE CATALOG, USE SCHEMA, and SELECT on the online tables’ catalog/schema. Databricks grants these automatically in most cases; if your governance is stricter, verify and grant them explicitly.

  Quick diagnostic checklist

  • Online tables show AVAILABLE in UC, created by publish_table to a DatabricksOnlineStore.
  • No SQL-store secrets/prefix env vars are present on the endpoint; nothing like PREFIX_USER/PASSWORD or LAKEBASE_USER/PASSWORD is set.
  • Model version was logged with FeatureEngineeringClient.log_model on a recent FE client/runtime; not a legacy Workspace Feature Store artifact.
  • The endpoint’s auto service principal has catalog/schema/table access needed to read the online tables.

  If you apply the steps above, the “_USER not found” path will no longer be taken and the endpoint will use OAuth-based access to Lakebase online tables automatically.

  # DBR 16.4 LTS ML (or serverless) + databricks-feature-engineering >= 0.13.0
  from databricks.feature_engineering import FeatureEngineeringClient, FeatureLookup
  from databricks.ml_features.entities.online_store import DatabricksOnlineStore
  
  fe = FeatureEngineeringClient()
  
  # Ensure offline table has PK + CDF, then publish to Lakebase
  online_store = fe.get_online_store(name="my-online-store")
  fe.publish_table(
      online_store=online_store,
      source_table_name="catalog.schema.nyc_taxi_pickup_features_spn",
      online_table_name="catalog.schema.nyc_taxi_pickup_features_spn_online",
  )
  
  # Log model with feature lookups for automatic online serving
  feature_lookups = [
      FeatureLookup(
          table_name="catalog.schema.nyc_taxi_pickup_features_spn",
          lookup_key="pickup_location_id",
          feature_names=["avg_wait_time", "pickup_demand_score"],
      )
  ]
  # Train on training_set.load_df(), then:
  fe.log_model(
      model=trained_model,
      artifact_path="model",
      registered_model_name="catalog.schema.nyc_taxi_model",
      feature_lookups=feature_lookups,
  )

[peternagy]

Thanks for the reply It is very useful and comprehensive.

I managed to find another solution to the problem so I wanted to share some additional details on this topic:
I was using 15.4 LTS ML Runtime, this could have caused the problem - I did not switch to 16.4 since that might have other breaking changes on other parts of my project. 

My solution was: I realized that in the container created by Databricks for the serving endpoint behind the scenes it installs a package 'databricks-feature-lookup==1.*' inside the container and for my usecase it defaulted to version 1.4 which is an older version. Instead for the automatic authentication to Online tables to work 'databricks-feature-lookup==1.9' is required. -> I have not found any documentation that mentioned this dependency so I wanted to share this - it might be useful to mention this somewhere in the docs as it was a bit hidden and hard to debug.

By packaging the model with this dependency fixed inside the FeatureEngineeringClient.log_model() we can ensure that the right version of this package gets installed inside the serving-endpoint's docker and we won't get the auth error again.

fe.log_model(

    model=model,

    artifact_path="model_packaged",

    flavor=mlflow.lightgbm,

    training_set=training_set,

    registered_model_name=model_name,

    pip_requirements=["databricks-feature-lookup==1.9.0"]

)