topic Vector Search index not indexing the whole Delta table in Get Started Discussions

Vector Search index not indexing the whole Delta table

Nicolaus — Mon, 08 Jul 2024 06:05:43 GMT

I have a Delta table that I’m trying to index but when I try to create a vector search index with either the UI or the Python SDK, it only indexes 1 row out of my 3000 rows. I have tried using different vector search endpoints.

I have verified the following:

I have a Unity Catalog enabled workspace.
Serverless compute is enabled.
My source table has Change Data Feed enabled.
I have CREATE TABLE privileges.

Has anyone encountered the same issue?

Re: Vector Search index not indexing the whole Delta table

Nicolaus — Thu, 11 Jul 2024 05:59:11 GMT

Thanks for the suggestions, @Retired_mod!

I tried optimizing my table and batching my data into 500 rows but it’s still not indexing my Delta table. I’m sure Ensure that Unity Catalog and serverless compute are properly configured as well.

What is the maximum number of rows that can be indexed in Databricks? Is there a way to combine multiple indexes in Databricks? What other indexing strategies are available in Databricks that could be more efficient?