I am trying to create a vector search index for a Delta Table using Azure OpenAI embeddings (text-embedding-3-large). The table contains 5000 chunks with approx. 1000 tokens each. The OpenAI embeddings are generated through a Databricks model serving endpoint which forwards the embedding requests to our Azure deployment.
The latency of the index creation is incredibly high. To embed just 5000 chunks the initial sync takes over an hour. If I extrapolate this to 5 Mio. chunks, embedding them would take over a month.
The deployment in Azure can process 350K tokens/min and should not be the limiting factor.
I am currently creating the index like this:
vsc.create_delta_sync_index(
endpoint_name=VECTOR_SEARCH_ENDPOINT_NAME,
index_name=vs_index_fullname,
source_table_name=source_table_fullname,
pipeline_type="TRIGGERED",
primary_key="id",
embedding_source_column='content'
embedding_model_endpoint_name='text-embedding-3-large'
)
My assumption for the slow speed is that the index calculates the embeddings row by row, without async Azure API calls, and only starts with the next row once the previous embedding is created.
Which options do I have for speeding up the index creation?
