using openai Api in Databricks without iterating rows

Alessandro — Fri, 19 Jan 2024 16:16:05 GMT

Hi to everyone,

I have a delta table with a column 'comment' I would like to add a new column 'sentiment', and I would like to calculate it using openai API.

I already know how to create a databricks endpoint to an external model and how to use it (using for example langchain) on a single comment or iterating the comments in a table.

I would like to know if there is a way to do it more efficiently using an udf, or if it is possible to create a model in Databricks that calls the openai API with a fixed prompt, then create an endpoint to that model and call that in an udf.

Thank you in advance for the response, I hope my question is clear, if not let me know and I will try to explain it better

topic using openai Api in Databricks without iterating rows in Machine Learning

using openai Api in Databricks without iterating rows