Vector search index stops at 45406

nikviz
New Contributor II

I am trying to create a vector search index for a table, but it stops at 45406 rows. I can see that the writeback table has all the records but the indexing stops. Is there a hard limit on index?

Walter_C
Databricks Employee
Databricks Employee

There are some limits that you can be hitting:

  • Row Size for Delta Sync Index: The maximum row size is 100KB.
  • Embedding Source Column Size for Delta Sync Index: The maximum size is 32764 bytes.
  • Bulk Upsert Request Size Limit for Direct Vector Index: The maximum size is 10MB.
  • Bulk Delete Request Size Limit for Direct Vector Index: This limit is not specified in the provided context.

 

 

View solution in original post

nikviz
New Contributor II

Thank you so much. I reduced my chunk size and it worked.