Databricks OpenAI Integration Issue

[leelee3000]

Facing challenges in leveraging Databricks for serving, logging, and monitoring OpenAI usage within an Azure environment. Specifically, encountering issues with Inference Tables not being enabled in the UI when creating serving endpoints with external providers. Implemented a workaround involving a custom PyFunc class and API calls for model registration and serving endpoint creation. Seeking insights or solutions to streamline the process and enable efficient logging and monitoring of OpenAI usage. Any guidance or expertise on this matter would be greatly appreciated

[Kaniz]

Hi @leelee3000, Here are some insights that might help:

Inference Tables: If Inference Tables are not being enabled in the UI when creating serving endpoint.... However, please note that once you disable inference tables on an endpoint, you cannot re-enable them. To continue using inference tables, you must create a new endpoint and enable inference tables on it.

Custom PyFunc Class: MLflow supports custom models of the mlflow.pyfunc flavor. You can create a custom class inherited from the mlflow.pyfunc.PythonModel that provides a predict f...

.

Model Registration and Serving Endpoint Creation: After logging your custom pyfunc model, you can re....

Logging and Monitoring: Azure OpenAI Service uses Azure Monitor, which provides out-of-box dashboard.... You can analyze metrics for your Azure OpenAI Service resources with Azure Monitor tools in the Azur.... For more comprehensive logging and monitoring, you might consider implementing solutions like the on....

[Debayan]

Hi, In addition to the previous message, you can refer to https://learn.microsoft.com/en-us/azure/databricks/large-language-models/ai-generate-text-example.