- 5270 Views
- 3 replies
- 1 kudos
Behavior of Vector Index Sync with Delta Tables When Using OVERWRITE vs MERGE in Databricks
I'm working with vector search in Databricks using vector index sync with Delta tables, and I'm a bit unclear on how updates to the source table affect the vector index, specifically when using different write operations.If I overwrite the source Del...
- 5270 Views
- 3 replies
- 1 kudos
- 1 kudos
From community experience, vector index sync behavior depends heavily on how the Delta table is updated. With OVERWRITE, the table is effectively replaced, so the vector index typically treats this as a full refresh. Existing embeddings are dropped a...
- 1 kudos
- 6000 Views
- 1 replies
- 0 kudos
How to perform combined search on structured and unstructured data in databrick using RAG or other
I created a RAG application in databricks which performs the following steps:1. Extract text from PDF files2. Prepare embeddings on extracted text and create vector search index3. Create a LLM model and served the model which can answer question base...
- 6000 Views
- 1 replies
- 0 kudos
- 0 kudos
You can achieve combined retrieval across both PDF-extracted unstructured data and multiple columns from structured Delta tables in Databricks, but there are important considerations and available patterns to optimize this workflow for your RAG appli...
- 0 kudos
- 3544 Views
- 1 replies
- 0 kudos
Resolved! Roadmap for vector_search function
I was wondering if there was a roadmap for the development of the vector_search function: vector_search function | Databricks DocumentationSpecifically, I was wondering if / when the following limitations may be lifted:Querying DIRECT_ACCESS index ty...
- 3544 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @jAAmes_bentley, DIRECT_ACCESS & filters_json are not currently supported with vector_search sql function. These are on our roadmap, but we don’t have concrete ETAs to share at the moment as we’re focusing on other high-priority tasks. Hybrid s...
- 0 kudos
- 1474 Views
- 1 replies
- 1 kudos
Is there a batch query option for similar vectors available in Mosaic AI Vector Search?
Hi all,I have been exploring databrick's vector search offering, and was wondering if it had the ability to run a batch query?I would like to be able to do something like the following where both embedding_1 and embedding_2 are separate vectors that ...
- 1474 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Cummings012 ,As per documentation it is not possible, as the API supports single queries.The workaround is to process these queries in parallel to improve throughput.
- 1 kudos
- 4599 Views
- 1 replies
- 0 kudos
Resolved! Vector Index Creation Initializing Phase
When using the web UI to create a vector index from an existing table with chunked data the creation fails at the "Initializing" phase after about 20 minutes. I have validated the requirements are satisfied that are detailed here.I have also seen th...
- 4599 Views
- 1 replies
- 0 kudos
- 0 kudos
I believe this "self-resolved". Even though I was technically on a premium plan the trial period had not completed. I think at this point I can say it is resolved.
- 0 kudos
- 3717 Views
- 1 replies
- 0 kudos
How to to create vector search index on multiple columns in the delta table
I want to create vector search index on a delta table using multiple "Embedding source column" from the table. I don't see an option to choose multiple columns so what's the alternative? create index per source column? so multiple indexes for sing...
- 3717 Views
- 1 replies
- 0 kudos