Hi to everyone,
I have a delta table with a column 'comment' I would like to add a new column 'sentiment', and I would like to calculate it using openai API.
I already know how to create a databricks endpoint to an external model and how to use it (using for example langchain) on a single comment or iterating the comments in a table.
I would like to know if there is a way to do it more efficiently using an udf, or if it is possible to create a model in Databricks that calls the openai API with a fixed prompt, then create an endpoint to that model and call that in an udf.
Thank you in advance for the response, I hope my question is clear, if not let me know and I will try to explain it better
