Vector Index format.

[skrishnaprasad]

On of the key benefits of a delta table format is that it's open.  Is this also the case for vector indexes as well? If so where could I find it's specification?

In Databricks today, we see that we can create and manage a vector index using API's. (https://docs.databricks.com/api/azure/workspace/vectorsearchindexes/createindex). Is this also possible using spark data frame operations as well?

thanks

Krishna

[Alberto_Umana]

The Delta table format is indeed open, which is one of its key benefits. However, the same cannot be said for vector indexes. The specifications for vector indexes are not openly available in the same way as Delta tables.

Regarding the creation and management of vector indexes, it is currently possible to do so using the Databricks UI, the Python SDK, and the REST API. The documentation does not mention the ability to create and manage vector indexes directly using Spark DataFrame operations.

For more detailed information on creating and managing vector indexes, you can refer to the following resources:

• How to create and query a vector search index
  Visit URL: https://docs.databricks.com/en/generative-ai/create-query-vector-search.html
• https://docs.databricks.com/api/azure/workspace/vectorsearchindexes/createindex

These resources provide comprehensive guidance on using the UI, Python SDK, and REST API for vector index operations.

[skrishnaprasad]

Thank you for your response. Is there a view on making these as open as the delta format? 

[TejaJuttu]

HI Team, I am trying to update an existing vector search index with new data which is in another delta table. But i have no luck figuring out how to do it using Python SDK. Can you please hekp point to the right resources?