using openai Api in Databricks without iterating rows

Machine Learning

Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.

Hi to everyone,

I have a delta table with a column 'comment' I would like to add a new column 'sentiment', and I would like to calculate it using openai API.

I already know how to create a databricks endpoint to an external model and how to use it (using for example langchain) on a single comment or iterating the comments in a table.

I would like to know if there is a way to do it more efficiently using an udf, or if it is possible to create a model in Databricks that calls the openai API with a fixed prompt, then create an endpoint to that model and call that in an udf.

Thank you in advance for the response, I hope my question is clear, if not let me know and I will try to explain it better