https://community.databricks.com/t5/data-engineering/generate-embeddings-from-third-party-api-in-delta-live-tables/m-p/71628#M34359 <P class="">Hi</P><P class="">We currently have a Delta Live Tables flow through which textual data is flowing. As a final enrichment step we would also want to generate embeddings using a third party api provider (probably <A href="https://www.voyageai.com/" target="_blank" rel="noopener"><SPAN class="">Voyage.AI</SPAN></A>). They support batch embedding which would greatly help speed this up (but we would still rate limits).</P><P class="">Is it best-practice to include this as a UDF in the DLT flow? Or should these be generated in another step?</P><P class="">Thanks</P><P class="">&nbsp;</P> Tue, 04 Jun 2024 14:58:29 GMT LucasBelpaire 2024-06-04T14:58:29Z https://community.databricks.com/t5/data-engineering/generate-embeddings-from-third-party-api-in-delta-live-tables/m-p/71628#M34359 <P class="">Hi</P><P class="">We currently have a Delta Live Tables flow through which textual data is flowing. As a final enrichment step we would also want to generate embeddings using a third party api provider (probably <A href="https://www.voyageai.com/" target="_blank" rel="noopener"><SPAN class="">Voyage.AI</SPAN></A>). They support batch embedding which would greatly help speed this up (but we would still rate limits).</P><P class="">Is it best-practice to include this as a UDF in the DLT flow? Or should these be generated in another step?</P><P class="">Thanks</P><P class="">&nbsp;</P> Tue, 04 Jun 2024 14:58:29 GMT https://community.databricks.com/t5/data-engineering/generate-embeddings-from-third-party-api-in-delta-live-tables/m-p/71628#M34359 LucasBelpaire 2024-06-04T14:58:29Z