Invoke Azure AI Search Endpoint through Databricks environment

[neha89]

Hi Team,

Is there a possibility to invoke Azure AI Search Vector DB endpoint(external) in Databricks environment based on input data in Databricks table.

Scenario: Client-Specific documents are already embedded in Azure AI Search Vector DB. Is there any way on how this existing Azure resource endpoint can be invoked inside Databricks for vector search rather than creating a new Vector DB in Databricks.

Any inputs/suggestions will be highly appreciated!

[mark_ott]

Yes, it is possible to invoke an Azure AI Search Vector DB endpoint from within a Databricks environment—allowing you to leverage your existing Azure resource for client-specific document retrieval, without needing to create a new vector database in Databricks itself.​

Concept and Workflow

• Azure AI Search supports REST and SDK APIs for vector search workloads, enabling external calls from other platforms, including Databricks.​

• To use it in Databricks, you can send HTTP requests (using libraries such as requests in Python or notebooks) to the Azure AI Search endpoint, passing your query vectors (for example, embeddings created from input data in your Databricks table).​

• The endpoint will return the most similar documents based on your query vector, making it possible to retrieve relevant client-specific content already embedded in Azure AI Search.

Suggested Approach

• Extract or compute embeddings for input data within Databricks (if required for your use-case).

• Invoke the Azure AI Search endpoint using Databricks' support for REST API calls, passing the query payload in the proper format (including vector query and any filters, as needed).​

• Process the search results in Databricks for downstream analytics or visualization.

Example Implementation Steps

• Use Python in a Databricks notebook to call Azure AI Search:

  • Install the required libraries: pip install requests.

  • Use the Azure AI Search REST API endpoint to perform a vector query (typically a POST request with the query vector and metadata).

  • Parse the results for further processing in Databricks.

Additional Notes

• The integration does not require building a new vector index or database in Databricks—you simply use Databricks as your orchestration and analytics layer, and Azure AI Search as your vector search backend.​

• This approach allows you to keep document storage and vector indexing centralized in Azure AI Search, supporting governance, scalability, and existing document embeddings.​

This solution provides a seamless bridge between Databricks and Azure AI Search for vector search operations, yielding scalable and flexible analytics workflows tailored to client-specific datasets stored in Azure AI Search.​