- 7159 Views
- 3 replies
- 1 kudos
Behavior of Vector Index Sync with Delta Tables When Using OVERWRITE vs MERGE in Databricks
I'm working with vector search in Databricks using vector index sync with Delta tables, and I'm a bit unclear on how updates to the source table affect the vector index, specifically when using different write operations.If I overwrite the source Del...
- 7159 Views
- 3 replies
- 1 kudos
- 1 kudos
From community experience, vector index sync behavior depends heavily on how the Delta table is updated. With OVERWRITE, the table is effectively replaced, so the vector index typically treats this as a full refresh. Existing embeddings are dropped a...
- 1 kudos
- 7287 Views
- 1 replies
- 0 kudos
How to perform combined search on structured and unstructured data in databrick using RAG or other
I created a RAG application in databricks which performs the following steps:1. Extract text from PDF files2. Prepare embeddings on extracted text and create vector search index3. Create a LLM model and served the model which can answer question base...
- 7287 Views
- 1 replies
- 0 kudos
- 0 kudos
You can achieve combined retrieval across both PDF-extracted unstructured data and multiple columns from structured Delta tables in Databricks, but there are important considerations and available patterns to optimize this workflow for your RAG appli...
- 0 kudos
- 4383 Views
- 1 replies
- 0 kudos
Resolved! Roadmap for vector_search function
I was wondering if there was a roadmap for the development of the vector_search function: vector_search function | Databricks DocumentationSpecifically, I was wondering if / when the following limitations may be lifted:Querying DIRECT_ACCESS index ty...
- 4383 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @jAAmes_bentley, DIRECT_ACCESS & filters_json are not currently supported with vector_search sql function. These are on our roadmap, but we don’t have concrete ETAs to share at the moment as we’re focusing on other high-priority tasks. Hybrid s...
- 0 kudos
- 1724 Views
- 1 replies
- 1 kudos
Is there a batch query option for similar vectors available in Mosaic AI Vector Search?
Hi all,I have been exploring databrick's vector search offering, and was wondering if it had the ability to run a batch query?I would like to be able to do something like the following where both embedding_1 and embedding_2 are separate vectors that ...
- 1724 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Cummings012 ,As per documentation it is not possible, as the API supports single queries.The workaround is to process these queries in parallel to improve throughput.
- 1 kudos
- 4866 Views
- 1 replies
- 0 kudos
Resolved! Vector Index Creation Initializing Phase
When using the web UI to create a vector index from an existing table with chunked data the creation fails at the "Initializing" phase after about 20 minutes. I have validated the requirements are satisfied that are detailed here.I have also seen th...
- 4866 Views
- 1 replies
- 0 kudos
- 0 kudos
I believe this "self-resolved". Even though I was technically on a premium plan the trial period had not completed. I think at this point I can say it is resolved.
- 0 kudos
- 4128 Views
- 1 replies
- 0 kudos
How to to create vector search index on multiple columns in the delta table
I want to create vector search index on a delta table using multiple "Embedding source column" from the table. I don't see an option to choose multiple columns so what's the alternative? create index per source column? so multiple indexes for sing...
- 4128 Views
- 1 replies
- 0 kudos