Migration from self-managed MLflow to Databricks managed MLflow.

Maverick1
Valued Contributor II

‎09-26-2021 11:34 PM

Would it be possible to somehow save the data, metrics of all experiments captured by self-managed mlflow using A/mazon RDS, S3 as backend and then load it to databricks managed mlflow and make it available in the UI? This is required as a part of migration activity.

Labels:

User16783853898
Databricks Employee
Databricks Employee

‎10-05-2021 11:27 AM

You can migrate your MLflow experiment and runs using the MLflow API with this toolkit: https://github.com/amesar/mlflow-export-import

Maverick1
Valued Contributor II
In response to User16783853898

‎03-07-2022 05:06 AM

@Andre Mesarovic​ : Is this solution, Databricks approved for enterprise usage?

I believe this is not an approved solution and also I can see many issues in it while doing import and export like nested metrics are not imported correctly etc.

User16783853898
Databricks Employee
Databricks Employee

‎03-07-2022 08:08 AM

mlflow-export-import is an open source tool used by both OSS and Databricks users. It is in the process of being a recommended solution for Databricks MLflow migrations. Create a ticket for any issues at: https://github.com/amesar/mlflow-export-import/issues

Maverick1
Valued Contributor II
In response to Retired_mod

‎05-20-2022 03:31 AM

@Kaniz Fatma​ : Hi Fatima,

mlflow-export-import is still not a recommended solution and has no support from Databricks. This is clarified by Databricks SA's themselves.

I have developed an another approach where I am capturing the statistics of a particular run which is needed to be migrated to higher env. (like pre-prod or prod) and then creating a mlflow run with those statistics instead of re-running the entire huge model. This is working perfectly and is within the logical support of Databricks.

View solution in original post