topic Re: Vector Index format. in Generative AI

Vector Index format.

skrishnaprasad — Thu, 05 Dec 2024 02:08:14 GMT

On of the key benefits of a delta table format is that it's open. Is this also the case for vector indexes as well? If so where could I find it's specification?

In Databricks today, we see that we can create and manage a vector index using API's. (https://docs.databricks.com/api/azure/workspace/vectorsearchindexes/createindex). Is this also possible using spark data frame operations as well?

thanks

Krishna

Re: Vector Index format.

Alberto_Umana — Fri, 06 Dec 2024 14:09:32 GMT

The Delta table format is indeed open, which is one of its key benefits. However, the same cannot be said for vector indexes. The specifications for vector indexes are not openly available in the same way as Delta tables.

Regarding the creation and management of vector indexes, it is currently possible to do so using the Databricks UI, the Python SDK, and the REST API. The documentation does not mention the ability to create and manage vector indexes directly using Spark DataFrame operations.

For more detailed information on creating and managing vector indexes, you can refer to the following resources:

These resources provide comprehensive guidance on using the UI, Python SDK, and REST API for vector index operations.

Re: Vector Index format.

skrishnaprasad — Sat, 07 Dec 2024 02:01:29 GMT

Thank you for your response. Is there a view on making these as open as the delta format?

Re: Vector Index format.

TejaJuttu — Wed, 21 May 2025 03:30:44 GMT

HI Team, I am trying to update an existing vector search index with new data which is in another delta table. But i have no luck figuring out how to do it using Python SDK. Can you please hekp point to the right resources?