cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 

databricks-vectorsearch lib install

ccataV
New Contributor

Hello,

I'm trying to create and query a vector searc index like in this example : How to create and query a Vector Search index | Databricks on AWS on a databricks on azure. I have a cluster ina  private network so i need to install the suggested lib : databricks-vectorsearch · PyPI manually . 

So i tried to find the right version for my cluster that is : 15.0 ML (includes Apache Spark 3.5.0, Scala 2.12) with Python version 3.10 but i dont manage , all are failing at installation . How i can find the right lib version to install it ?

Anyone had the same experience or faced something similar?

Any suggestion or help will be much appreciate.

Thank you. 

3 REPLIES 3

Kaniz
Community Manager
Community Manager

Hi @ccataVCreating and querying a vector search index using Databricks Vector Search is a powerful capability. Let’s break down the steps to achieve this:

  1. Create a Vector Search Endpoint:

  2. Configure an Endpoint for the Embedding Model (Optional):

    • If you want Databricks to compute the embeddings, set up a model serving endpoint to serve the embedding model.
    • Remove the default selection of “Scale to zero” when configuring the embedding endpoint.
    • For example notebooks, explore the Notebook examples for calling an embeddings model.
  3. Library Installation:

  4. Additional Considerations:

    • Ensure that your workspace has Unity Catalog enabled and serverless compute enabled.
    • The source table must have Change Data Feed enabled.
    • You’ll need appropriate privileges to create indexes and query endpoints.
    • Consider using service principal tokens instead of personal access tokens for authentication.

Remember that Databricks Vector Search is a powerful tool for managing vector indices, and it’s essential to follow the right steps and version compatibility to make the most of it. 

Best of luck with your vector search endeavors! 🚀

 

Kumaran
Valued Contributor III
Valued Contributor III

Hi @ccataV 

Thank you for contacting Databricks community support!

Can you try to use 14.3 ML DBR and see how it works?

shan_chandra
Esteemed Contributor
Esteemed Contributor

@ccataV - Databricks vector search is in public preview in selected regions as per the below documentation. However, since you look like a PVC user, you may need to reach out to Databricks support for accessibility.

https://docs.databricks.com/en/generative-ai/vector-search.html#limitations