- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-11-2025 05:22 AM
Hello All,
I have dataframe with 5 million rows and before we can setup vector search endpoint against index, we want to generate embeddings column for each of those rows. Please suggest whats an optimal way to do this?
We are in development phase so we need to do full load but later we will need to do same for incremental load.
Thanks & Regards,
Vikram
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-11-2025 08:08 AM
The easiest and most reliable way to generate embeddings for millions of rows is to let Databricks Vector Search compute them automatically during synchronization from a Delta table.
Vector Search can generate embeddings for you, keep them updated when new records are inserted or updated, and handle batching, scaling, and retries behind the scenes.
You don’t have to manually loop over rows or call a model serving endpoint, Vector Search handles that for you.
https://learn.microsoft.com/en-us/azure/databricks/generative-ai/create-query-vector-search
Handles full backfill (5M+ rows) efficiently
Supports incremental updates automatically via Delta change data
No manual code or loops required
Fully managed and Unity Catalog–governed
