topic Re: Vector search index initialization very slow in Machine Learning

Vector search index initialization very slow

RodrigoE — Wed, 17 Dec 2025 15:49:49 GMT

Hello,

I am creating a vector search index and selected Compute embeddings for a delta table with 19M records. Delta table has only two columns: ID (selected as index) and Name (selected for embedding). Embedding model is databricks-gte-large-en.

Index initialization estimated time to complete is 15 days!

I'm looking for advice to speed up initialization.

Thank you,

Rodrigo Escamilla

Re: Vector search index initialization very slow

iyashk-DB — Wed, 17 Dec 2025 15:55:55 GMT

Hi Rodrigo,

The issue that you are seeing is because these embeddings are computed on the Databricks-GTE-Large-EN endpoint, which is a Pay-Per-Token endpoint. These have very high latency when used. So if speed is a concern, we suggest you use the models present in system.ai schema and create a Provisioned Throughput endpoint with a larger number of Model Units to have higher throughput and faster computations of embeddings. Then use that endpiont for computing the embeddings.

Re: Vector search index initialization very slow

RodrigoE — Wed, 17 Dec 2025 15:58:37 GMT

Thank you very much for the fast response. I am new to databricks (and vector search). How do I go about "use the models present in system.ai schema and create a Provisioned Throughput endpoint with a larger number of Model Units"

Thank you,

Rodrigo Escamilla

Re: Vector search index initialization very slow

iyashk-DB — Wed, 17 Dec 2025 16:04:52 GMT

@RodrigoE please follow this document - https://docs.databricks.com/aws/en/machine-learning/foundation-model-apis/deploy-prov-throughput-foundation-model-apis#create-your-provisioned-throughput-endpoint-using-the-ui

Re: Vector search index initialization very slow

RodrigoE — Thu, 18 Dec 2025 14:33:44 GMT

Your recommendation addressed the issue. Followed the instructions and index initialization took only 8 hours - thank you!

Re: Vector search index initialization very slow

BadrErraji — Sun, 15 Mar 2026 11:57:56 GMT

Why the deltaSync doesn't compute the embedding in parralel instead of sequential.
That a major gap in the architecture no ?