Root cause in plain English

• The lookup client is trying to read SQL-style credentials like PREFIX_USER/PREFIX_PASSWORD for a third‑party online store and the “prefix” is empty, so it searches for “_USER” and fails. That auth scheme applies only to third‑party stores; Databricks Online Feature Store (Lakebase) does not require USER/PASSWORD/HOST env vars for serving.
   

Resolve in 6 steps

1) Use the Databricks Online Feature Store path end‑to‑end

• Make sure your feature tables are published to a Databricks Online Feature Store (Lakebase) via FeatureEngineeringClient.publish_table and that the online tables are AVAILABLE in UC. Don’t pass SQL OnlineStoreSpec or secret prefixes; just use DatabricksOnlineStore and let Databricks manage auth.
   

2) Confirm platform and client versions

• Use DBR 16.4 LTS ML (or Serverless) for notebooks/jobs that publish tables and log the model, and install databricks-feature-engineering >= 0.13.0 in those workflows. This is the required stack for Online Feature Stores and automatic lookup.
   
• If your model was logged on older clients/runtimes, re‑log it with the current databricks-feature-engineering so the serving image uses the modern lookup client (databricks-feature-lookup) and understands Lakebase metadata correctly.
   

3) Re‑publish feature tables correctly (if in doubt, redo)

• Before publishing: ensure primary key exists, PK columns are NOT NULL, and CDF is enabled on the offline table.
   
• Publish with DatabricksOnlineStore (no secrets/host/user/password). Wait for the online table status to be AVAILABLE in UC.
   

4) Re‑log and redeploy the model for automatic lookup

• Log via FeatureEngineeringClient.log_model with your FeatureLookup definitions. Models logged this way automatically track lineage and perform feature lookups from online stores at inference; no extra credentials are needed for Databricks Online Feature Store.
   

5) Remove third‑party store credentials and env vars

• On the serving endpoint, remove environment variables intended for SQL stores (for example, any PREFIX_USER/PREFIX_PASSWORD, LAKEBASE_USER/LAKEBASE_PASSWORD). These are documented for third‑party stores and will misroute the lookup client; Lakebase uses OAuth transparently and does not need them.
   

6) Validate endpoint permissions for the auto service principal

• Model Serving creates its own system service principal to query your online tables. Ensure it has USE CATALOG, USE SCHEMA, and SELECT on the online tables’ catalog/schema. Databricks grants these automatically in most cases; if your governance is stricter, verify and grant them explicitly.

  Quick diagnostic checklist

  • Online tables show AVAILABLE in UC, created by publish_table to a DatabricksOnlineStore.
     
  • No SQL-store secrets/prefix env vars are present on the endpoint; nothing like PREFIX_USER/PASSWORD or LAKEBASE_USER/PASSWORD is set.
     
  • Model version was logged with FeatureEngineeringClient.log_model on a recent FE client/runtime; not a legacy Workspace Feature Store artifact.
     
  • The endpoint’s auto service principal has catalog/schema/table access needed to read the online tables.
     

  If you apply the steps above, the “_USER not found” path will no longer be taken and the endpoint will use OAuth-based access to Lakebase online tables automatically.

  # DBR 16.4 LTS ML (or serverless) + databricks-feature-engineering >= 0.13.0
  from databricks.feature_engineering import FeatureEngineeringClient, FeatureLookup
  from databricks.ml_features.entities.online_store import DatabricksOnlineStore
  
  fe = FeatureEngineeringClient()
  
  # Ensure offline table has PK + CDF, then publish to Lakebase
  online_store = fe.get_online_store(name="my-online-store")
  fe.publish_table(
      online_store=online_store,
      source_table_name="catalog.schema.nyc_taxi_pickup_features_spn",
      online_table_name="catalog.schema.nyc_taxi_pickup_features_spn_online",
  )
  
  # Log model with feature lookups for automatic online serving
  feature_lookups = [
      FeatureLookup(
          table_name="catalog.schema.nyc_taxi_pickup_features_spn",
          lookup_key="pickup_location_id",
          feature_names=["avg_wait_time", "pickup_demand_score"],
      )
  ]
  # Train on training_set.load_df(), then:
  fe.log_model(
      model=trained_model,
      artifact_path="model",
      registered_model_name="catalog.schema.nyc_taxi_model",
      feature_lookups=feature_lookups,
  )