cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Vector search index stops at 45406

nikviz
New Contributor

I am trying to create a vector search index for a table, but it stops at 45406 rows. I can see that the writeback table has all the records but the indexing stops. Is there a hard limit on index?

1 ACCEPTED SOLUTION

Accepted Solutions

Walter_C
Databricks Employee
Databricks Employee

There are some limits that you can be hitting:

  • Row Size for Delta Sync Index: The maximum row size is 100KB.
  • Embedding Source Column Size for Delta Sync Index: The maximum size is 32764 bytes.
  • Bulk Upsert Request Size Limit for Direct Vector Index: The maximum size is 10MB.
  • Bulk Delete Request Size Limit for Direct Vector Index: This limit is not specified in the provided context.

 

 

View solution in original post

2 REPLIES 2

Walter_C
Databricks Employee
Databricks Employee

There are some limits that you can be hitting:

  • Row Size for Delta Sync Index: The maximum row size is 100KB.
  • Embedding Source Column Size for Delta Sync Index: The maximum size is 32764 bytes.
  • Bulk Upsert Request Size Limit for Direct Vector Index: The maximum size is 10MB.
  • Bulk Delete Request Size Limit for Direct Vector Index: This limit is not specified in the provided context.

 

 

Thank you so much. I reduced my chunk size and it worked.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group