Root cause in plain English
- The lookup client is trying to read SQL-style credentials like PREFIX_USER/PREFIX_PASSWORD for a third‑party online store and the “prefix” is empty, so it searches for “_USER” and fails. That auth scheme applies only to third‑party stores; Databricks Online Feature Store (Lakebase) does not require USER/PASSWORD/HOST env vars for serving.
Resolve in 6 steps
1) Use the Databricks Online Feature Store path end‑to‑end
- Make sure your feature tables are published to a Databricks Online Feature Store (Lakebase) via FeatureEngineeringClient.publish_table and that the online tables are AVAILABLE in UC. Don’t pass SQL OnlineStoreSpec or secret prefixes; just use DatabricksOnlineStore and let Databricks manage auth.
2) Confirm platform and client versions
- Use DBR 16.4 LTS ML (or Serverless) for notebooks/jobs that publish tables and log the model, and install databricks-feature-engineering >= 0.13.0 in those workflows. This is the required stack for Online Feature Stores and automatic lookup.
- If your model was logged on older clients/runtimes, re‑log it with the current databricks-feature-engineering so the serving image uses the modern lookup client (databricks-feature-lookup) and understands Lakebase metadata correctly.
3) Re‑publish feature tables correctly (if in doubt, redo)
- Before publishing: ensure primary key exists, PK columns are NOT NULL, and CDF is enabled on the offline table.
- Publish with DatabricksOnlineStore (no secrets/host/user/password). Wait for the online table status to be AVAILABLE in UC.
4) Re‑log and redeploy the model for automatic lookup
- Log via FeatureEngineeringClient.log_model with your FeatureLookup definitions. Models logged this way automatically track lineage and perform feature lookups from online stores at inference; no extra credentials are needed for Databricks Online Feature Store.
5) Remove third‑party store credentials and env vars
- On the serving endpoint, remove environment variables intended for SQL stores (for example, any PREFIX_USER/PREFIX_PASSWORD, LAKEBASE_USER/LAKEBASE_PASSWORD). These are documented for third‑party stores and will misroute the lookup client; Lakebase uses OAuth transparently and does not need them.
6) Validate endpoint permissions for the auto service principal
- Model Serving creates its own system service principal to query your online tables. Ensure it has USE CATALOG, USE SCHEMA, and SELECT on the online tables’ catalog/schema. Databricks grants these automatically in most cases; if your governance is stricter, verify and grant them explicitly.
Quick diagnostic checklist
- Online tables show AVAILABLE in UC, created by publish_table to a DatabricksOnlineStore.
- No SQL-store secrets/prefix env vars are present on the endpoint; nothing like PREFIX_USER/PASSWORD or LAKEBASE_USER/PASSWORD is set.
- Model version was logged with FeatureEngineeringClient.log_model on a recent FE client/runtime; not a legacy Workspace Feature Store artifact.
- The endpoint’s auto service principal has catalog/schema/table access needed to read the online tables.
If you apply the steps above, the “_USER not found” path will no longer be taken and the endpoint will use OAuth-based access to Lakebase online tables automatically.
# DBR 16.4 LTS ML (or serverless) + databricks-feature-engineering >= 0.13.0
from databricks.feature_engineering import FeatureEngineeringClient, FeatureLookup
from databricks.ml_features.entities.online_store import DatabricksOnlineStore
fe = FeatureEngineeringClient()
# Ensure offline table has PK + CDF, then publish to Lakebase
online_store = fe.get_online_store(name="my-online-store")
fe.publish_table(
online_store=online_store,
source_table_name="catalog.schema.nyc_taxi_pickup_features_spn",
online_table_name="catalog.schema.nyc_taxi_pickup_features_spn_online",
)
# Log model with feature lookups for automatic online serving
feature_lookups = [
FeatureLookup(
table_name="catalog.schema.nyc_taxi_pickup_features_spn",
lookup_key="pickup_location_id",
feature_names=["avg_wait_time", "pickup_demand_score"],
)
]
# Train on training_set.load_df(), then:
fe.log_model(
model=trained_model,
artifact_path="model",
registered_model_name="catalog.schema.nyc_taxi_model",
feature_lookups=feature_lookups,
)
