databricks-vectorsearch lib install

[ccataV]

Hello,

I'm trying to create and query a vector searc index like in this example : How to create and query a Vector Search index | Databricks on AWS on a databricks on azure. I have a cluster ina  private network so i need to install the suggested lib : databricks-vectorsearch · PyPI manually . 

So i tried to find the right version for my cluster that is : 15.0 ML (includes Apache Spark 3.5.0, Scala 2.12) with Python version 3.10 but i dont manage , all are failing at installation . How i can find the right lib version to install it ?

Anyone had the same experience or faced something similar?

Any suggestion or help will be much appreciate.

Thank you. 

[Kaniz_Fatma]

Hi @ccataV, Creating and querying a vector search index using Databricks Vector Search is a powerful capability. Let’s break down the steps to achieve this:

1. Create a Vector Search Endpoint:

   • You can create a vector search endpoint using the Databricks UI, Python SDK, or the REST API.
   • In the UI, navigate to Compute, click the Vector Search tab, and then click Create. Provide a name for the endpoint and confirm.
   • Using the Python SDK, you can use the create_endpoint() function to create an endpoint¹.
   • For REST API details, refer to the official documentation.

2. Configure an Endpoint for the Embedding Model (Optional):

   • If you want Databricks to compute the embeddings, set up a model serving endpoint to serve the embedding model.
   • Remove the default selection of “Scale to zero” when configuring the embedding endpoint.
   • For example notebooks, explore the Notebook examples for calling an embeddings model.

3. Library Installation:

   • To manually install the databricks-vectorsearch library, ensure you have the correct version for your cluster.
   • You mentioned using Databricks Runtime 15.0 with Python 3.10. However, note that Databricks Runtime 15.0 actually comes with Python 3.11.0².
   • For the library version, refer to the official Databricks Runtime release notes.
   • If you encounter installation failures, consider checking the compatibility of the library with your specific environment.

4. Additional Considerations:

   • Ensure that your workspace has Unity Catalog enabled and serverless compute enabled.
   • The source table must have Change Data Feed enabled.
   • You’ll need appropriate privileges to create indexes and query endpoints.
   • Consider using service principal tokens instead of personal access tokens for authentication.

Remember that Databricks Vector Search is a powerful tool for managing vector indices, and it’s essential to follow the right steps and version compatibility to make the most of it. 

Best of luck with your vector search endeavors! 🚀

[Kumaran]

Hi @ccataV 

Thank you for contacting Databricks community support!

Can you try to use 14.3 ML DBR and see how it works?

[shan_chandra]

@ccataV - Databricks vector search is in public preview in selected regions as per the below documentation. However, since you look like a PVC user, you may need to reach out to Databricks support for accessibility.

https://docs.databricks.com/en/generative-ai/vector-search.html#limitations