Knowledge Assistant fails to load after indexing; generated AI Search index cannot be reused

[qduan]

Hi Databricks Community,

I created a Knowledge Assistant through Databricks → Agents, using files as the knowledge source. The embedding and indexing were handled automatically during setup.

Since then, I have been unable to open the assistant. The UI displays:

We could not load the agent

Please try again later, or contact support.

Workaround attempted

To continue working, I tried creating another Knowledge Assistant and selecting the existing AI Search index as its knowledge source. However, the index did not appear as an available option.

I asked Genie to investigate. It reported that the automatically created index, [index_name], uses databricks-agent-bricks-embedding-v1, while Knowledge Assistant supports indexes using only:

• databricks-gte-large-en
• databricks-bge-large-en
• databricks-qwen3-embedding-0-6b

This is Genie's explanation; I have not independently confirmed that this restriction is the cause.

Expected behavior

The original Knowledge Assistant should remain accessible after indexing. Alternatively, I would expect to be able to reuse its generated index in a new assistant without embedding and indexing the same files again.

Questions

1. How can I diagnose and resolve the “We could not load the agent” error? Are there logs I can check?
2. Is an index using databricks-agent-bricks-embedding-v1 intentionally unavailable as a source for another Knowledge Assistant?
3. Is there a supported way to recover the original assistant or reuse its existing index without re-embedding the source files?

Environment

• Cloud provider / region: Azure/EMEA
• Approximate creation time and time zone: 11-Sep, around 12am Amsterdam time
• Source file types and approximate volume: more than 2000 files, mixed types, PDFs, pptx, etc.

Any guidance on recovery or index reuse would be appreciated.

[Louis_Frolio]

Hello @qduan , I did some digging and here is what I found.

Genie has the embedding piece right. The Knowledge Assistant docs list exactly three supported models for a bring-your-own AI Search index: databricks-gte-large-en, databricks-bge-large-en, and databricks-qwen3-embedding-0-6b. An index built automatically from a files source uses a Databricks-managed default model that isn't on that list, so it won't show up in the picker for a new assistant. That's documented behavior, not a bug on your end. I couldn't find anything public that promises reuse of that generated index without re-embedding, so I'd treat it as unsupported today.

That restriction doesn't explain the "We could not load the agent" error, though. Those are two separate problems, and the load failure needs actual evidence. On logs and diagnosis:

1. Query the audit log system table for service_name = knowledgeAssistant. Assistant and knowledge-source create, update, and sync events are logged there, so a failed or stuck sync should be visible.
2. Call the Knowledge Assistants API (SDK or REST) to get the agent and list its sources. If the API returns a status, you've bypassed whatever the UI is choking on.
3. In Catalog Explorer, check the generated index and its AI Search endpoint. Under Serving, check the agent's endpoint. Either one offline or failed will break the agent page.
4. Confirm the original volume still exists and your user still has access to it.
5. Open browser dev tools, reload the agent page, and grab the failing request and response from the Network tab. That payload is far more specific than the UI message.

Then open a Databricks Support case with the assistant ID, workspace ID, region, failure time, and whatever the audit log and network call gave you. A generic load error alone isn't enough to tell a permissions problem from an ingestion or service-side one. Don't delete the assistant or its source while this is open. Deleting an assistant removes everything associated with it from default storage.

If support can't revive it, you're looking at re-indexing, either a new assistant from the same files or your own AI Search index using one of the three supported models. The upside of building your own is that multiple assistants can share it and it updates automatically with no manual sync. One thing to check with 2000+ mixed files: anything over 100 MB, or over 500 pages for PDF, DOC, and PPT (each slide counts as a page), is skipped during ingestion. If the original build tripped on something in that pile, it could be part of the story.

References:

• Knowledge Assistant: https://learn.microsoft.com/en-us/azure/databricks/agents/agent-bricks/knowledge-assistant
• Create AI Search endpoints and indexes: https://learn.microsoft.com/en-us/azure/databricks/ai-search/create-ai-search
• Audit log reference (knowledgeAssistant events): https://learn.microsoft.com/en-us/azure/databricks/admin/account-settings/audit-logs
• Knowledge Assistants API: https://docs.databricks.com/api/workspace/knowledgeassistants
• Databricks Help Center: https://help.databricks.com/s/

Regards, Louis.