Create Databricks Dashboards on MLFlow Metrics

[rahuja]

Hello

Currently we have multiple ML models running in Production which are logging metrics and other meta-data on mlflow. I wanted to ask is it possible somehow to build Databricks dashboards on top of this data and also can this data be somehow available as databricks tables to do some analysis on it?    

#machinelearning #mlflow #databricks

[Kaniz_Fatma]

Hi @rahuja,

You can create dashboards in Databricks using MLflow data. To achieve this, you can leverage the mlflow.search_runs API to pull aggregate metrics from your MLflow runs and display them in a custom dashboard1. Regularly reviewing these metrics can provide insight into your progress and productivity. For example, you can track improvements in goal metrics like revenue or accuracy over time, across multiple runs and experiments. Additionally, you can use MLflow data to create Databricks tables for further analysis.

If you need more details or assistance, feel free to ask! 😊

[rahuja]

Hello @Kaniz_Fatma 
Thanks for responding. I think you  are talking about using the Python API. But we don't want that is it possible since MLFlow also uses an sql table to store metrics. To expose those tables as a part of our meta-store and build dashboard on top of it? We don't want to use compute for accessing data that is already stored in some DB in the backend.