Generate embeddings for 50 million rows in dataframe

vikram_p — Tue, 11 Nov 2025 13:22:56 GMT

Hello All,

I have dataframe with 5 million rows and before we can setup vector search endpoint against index, we want to generate embeddings column for each of those rows. Please suggest whats an optimal way to do this?

We are in development phase so we need to do full load but later we will need to do same for incremental load.

Thanks & Regards,

Vikram

Re: Generate embeddings for 50 million rows in dataframe

bianca_unifeye — Tue, 11 Nov 2025 16:08:42 GMT

The easiest and most reliable way to generate embeddings for millions of rows is to let Databricks Vector Search compute them automatically during synchronization from a Delta table.
Vector Search can generate embeddings for you, keep them updated when new records are inserted or updated, and handle batching, scaling, and retries behind the scenes.

You don’t have to manually loop over rows or call a model serving endpoint, Vector Search handles that for you.

https://learn.microsoft.com/en-us/azure/databricks/generative-ai/create-query-vector-search

Handles full backfill (5M+ rows) efficiently
Supports incremental updates automatically via Delta change data
No manual code or loops required
Fully managed and Unity Catalog–governed

topic Re: Generate embeddings for 50 million rows in dataframe in Data Engineering

Generate embeddings for 50 million rows in dataframe

Re: Generate embeddings for 50 million rows in dataframe