databricks-vectorsearch 0.53 unable to use similarity_search()

snaveedgm
New Contributor

‎03-21-2025 09:22 AM - edited ‎03-21-2025 09:25 AM

I have an issue with databricks-vectorsearch package. Version 0.51 suddenly stopped working this week because:

  • It now expected me to provide azure_tenant_id in addition to service principal's client ID and secret.
  • After supplying tenant ID, it showed some bug like "Unsupported Types" time.time() + oauth_token_data["expires_in"].
  • I then upgraded to 0.53 and this bug was resolved and some functions like get_index() started working. However, now the similarity_search() function gives:
    • b'{"error_code":"PERMISSION_DENIED","message":"Failed to call Model Serving endpoint: databricks-bge-large-en."}', status_code 403
  • I have tried querying the databricks-bge-large-en  endpoint separately via REST for my service account and it works fine, so the issue seems specific to this package.

Please advise.

 

Code Snippet:

from databricks.vector_search.client import VectorSearchClient

 

vsc = VectorSearchClient(

    workspace_url=workspace_url,

    service_principal_client_id=sp_client_id,

    service_principal_client_secret=sp_client_secret,

    azure_tenant_id=tenant_id

)

index = vsc.get_index(endpoint_name=endpoint_name, index_name=index_name)

index.similarity_search(num_results=3, columns=["chunked_text"], query_text="example_query")

0 Kudos

stbjelcevic
Databricks Employee
Databricks Employee

‎10-28-2025 08:58 AM

Hi @snaveedgm ,

This is interesting - can you double-check that the service principal has CAN QUERY on the embedding endpoint used for ingestion and/or querying (databricks-bge-large-en in your case)? Even though your direct REST test works, double-check permissions are consistent across workspaces/endpoints if you have multi-workspace setups.

Given your environment and error, the fastest unblock is either PAT or query_vector:

from databricks.vector_search.client import VectorSearchClient

# Use PAT (set DATABRICKS_TOKEN in the environment or pass personal_access_token)
vsc = VectorSearchClient(
workspace_url=workspace_url,
personal_access_token=os.environ["DATABRICKS_TOKEN"] # or pass the token directly
)

index = vsc.get_index(endpoint_name=endpoint_name, index_name=index_name)