Model Serving An internal error occurred during feature store lookup all deploys failing

[ticusss]

We can't deploy models to Model Serving. Failures started around 2026-08-16 and were intermittent at first — our current production config deployed cleanly on 08-20 — but since then every attempt fails at the feature store lookup setup step, with no diagnostics we can find anywhere. Hoping someone has seen this.

Failure mode 1 — fails after linking everything

Endpoint events look like this every time:

Endpoint updated by <user>
[25-34x] Linked with Databricks Online Store table ...
System service principal creation with ID `<uuid>` succeeded
System role ... created successfully for Online Feature Store `<store>/production`
An internal error occurred during feature store lookup setup <-- ~31-126s later
Served entity `<model>` entered DEPLOYMENT_FAILED

Served entity message: Online feature store setup failed. Please see event logs for...

It never reaches "Provisioning resources". serving-endpoints logs returns "There are currently no replicas in a running state", and build logs show a clean container build. So no user code ever runs and there is no traceback on our side.

The same model on the same 25 tables also showt fails. Nothing changed on our side betweenthese two deploys:
- 08-20, succeeded: update to system service principal created in 40 seconds, DEPLOYMENT_READY after 4m49s total.
- 08-27, failed: update to system service principal created in 8m27s, internal error at 9m52s.

Failure mode 2 — update hangs in pending with zero events
Separately, some updates never fail or progres a pending_config in DEPLOYMENT_CREATING, served entity message Creating resources for served entity, and no events are emitted at all — not even a single Linked with Databricks Online Store table line. We've left these running for 4h48m and 2h18m before cancelling by hand. Six consecutive attempts on one endpoint, zero tables linked on any of them.

Two things about this that may be relevant:

- One variant involved an online store whose Lakebase project was created through the Lakebase UI rather than create_online_store. publish_table against it succeeded, the online tables are ACTIVE and syncing, but no endpoint has ever linked a single one of them. The project does not appear in fe.list_online_stores(), which the docs say is expected for UI-created projects.
- A wedged update degrades everything else. With two stuck updates still pending, table linking on unrelated endpoints took 49–76 minutes; on a clear queue the same work takes under 90 seconds. This makes failures look far more random than they are, so we now cancel every failed or stuck update before measuring anything.

Setup

- ~34 feature tables with FeatureLookup, logged via FeatureEngineeringClient.log_model
- Online Feature Store created 2025-10-20 as Lakebase Provisioned, CU_1. Per the upgrade docs, all Provisioned instances were migrated to the Autoscaling platform in July 2026; the Database instance API still reports ours as capacity: CU_1, PG_VERSION_16 - Online tables are SYNCED_TABLE_ONLINE_NO_PENnity_catalog_provisioning_state: ACTIVE, syncingnormally
- Container resolves databricks-feature-lookup==1.* → 1.14.0, with mlflow==3.0.0 - databricks-feature-engineering==0.13.0 for p

Already ruled out (please don't suggest these)

- FEATURE_SERVING_CONNECTION_POOL_SIZE does not fix this. We tried 2, 3, 4 and 10. Deploys failed at every value, and the values that once looked promising were jusmade no difference.
- Not table count — a 25-table model that deployed cleanly on 08-20 now fails identically. 34-table models fail the same way.
- Not the model artifact — the same model version deployed fine on one endpoint while failing on another, with byte-identical build logs.
- Not tables unpublished / source_table_id mismatch — the two documented causes of "No suitable online store found for feature tables". We get a different, unnamed error and no table is ever named in it.
- Not concurrent sync — we checked every sync pipeline against a failure window. The sync wave started two minutes after the failure; nothing was syncing during the deploy.
- Not catalog/database name mismatch — catalog name and effective_logical_database_name match.
- Not connection capacity — max_connections 1802, ~124 in use, role rolconnlimit -1.
- Not scale-to-zero — docs confirm it isn't enabled by default on upgraded instances.
- Not grants, schemas, or keys — offline and online schemas match, PKs non-null, timeseries keys set, grants identical between tables that work and tables that don't.
- Not a documented limit — 50 tables allowed per model, we use 34. FeatureSpec well under the 1000-feature cap.
- Not DBR version, workload size, or container libraries.

The only two failures that left an audit trace

Across dozens of failures, exactly two show anything in system.access.audit (action_name='getOnlineFeatureTables', non-200 status):

RESOURCE_EXHAUSTED: You've exceeded the rate limit
INTERNAL_ERROR: Failed to fetch database instance with ID <online-store-instance-uuid> Every other failure produces no audit row at areTables call surrounding them returns 200. Sowhatever fails does not write to the audit table.

Questions
1. The one traced error we have is Failed to fetch database instance with ID ..., which is the UUID of our online stor— an instance that per the docs was migrated faling in July. We are not claiming the upgradebroke this, since deploys worked fine for week error expected against an upgraded instance,and could the lookup setup path resolve it inconsistently?
2. Is An internal error occurred during feature store lookup setup a catch-all? Is there any log surface that cits real cause, given it produces no replica l3. For failure mode 2 — is there any way to sewhen it emits no events? And is there asupported way to detect a wedged update, short of waiting hours and cancelling manually? 4. We see 289 orphaned Postgres roles in the omed role per deploy attempt, never reclaimedwhen a served entity or endpoint config is tor Lakebase Autoscaling has a 500-roles-per-branch limit that Provisioned did not. Is this accumulation expected, and is manual cleanup safe while an endpoint is serving?
5. Has anyone recovered from this? Or is standonline store with create_online_store andrepublishing everything the only route?

[AbhilashNagilla]

1. The upgrade documentation says existing feature and model serving workloads continue "with no action required" and the create_online_store and update_online_store interfaces need no changes. It doesn't tie Failed to fetch database instance with ID ... to the migration, so that error alone doesn't establish the migration as its cause.

2. Neither the Online Feature Store troubleshooting section nor the separate Feature Store troubleshooting page lists An internal error occurred during feature store lookup setup, defines it as a catch-all, or identifies another log surface for it.

3. The Model Serving debugging guide says to contact Support if no build event or message appears within one hour of creating an endpoint; it doesn't explicitly extend that threshold to updates. The 30/60/120-minute deployment timeouts start after the container builds, so they don't govern this earlier Creating resources hang.

4. The upgrade documentation confirms a 500-role-per-branch limit on Autoscaling that Provisioned didn't have. It doesn't say whether service-principal roles left by failed deploys count toward that limit, whether their accumulation is expected, or whether deleting them while an endpoint is serving is safe.

5. The Online Feature Store documentation describes create_online_store followed by publish_table as the setup path; it doesn't establish rebuilding as the only recovery for this error.

I would open one Support case with the traced instance UUID, the affected endpoint IDs, and the role count before rebuilding the store or deleting roles.