topic Re: Migration from self-managed MLflow to Databricks managed MLflow. in Machine Learning

Migration from self-managed MLflow to Databricks managed MLflow.

Maverick1 — Mon, 27 Sep 2021 06:34:10 GMT

Would it be possible to somehow save the data, metrics of all experiments captured by self-managed mlflow using A/mazon RDS, S3 as backend and then load it to databricks managed mlflow and make it available in the UI? This is required as a part of migration activity.

Re: Migration from self-managed MLflow to Databricks managed MLflow.

User16783853898 — Tue, 05 Oct 2021 18:27:30 GMT

You can migrate your MLflow experiment and runs using the MLflow API with this toolkit: https://github.com/amesar/mlflow-export-import

Re: Migration from self-managed MLflow to Databricks managed MLflow.

Maverick1 — Mon, 07 Mar 2022 13:06:27 GMT

@Andre Mesarovic : Is this solution, Databricks approved for enterprise usage?

I believe this is not an approved solution and also I can see many issues in it while doing import and export like nested metrics are not imported correctly etc.

Re: Migration from self-managed MLflow to Databricks managed MLflow.

User16783853898 — Mon, 07 Mar 2022 16:08:00 GMT

mlflow-export-import is an open source tool used by both OSS and Databricks users. It is in the process of being a recommended solution for Databricks MLflow migrations. Create a ticket for any issues at: https://github.com/amesar/mlflow-export-import/issues

Re: Migration from self-managed MLflow to Databricks managed MLflow.

Maverick1 — Fri, 20 May 2022 10:31:26 GMT

@Kaniz Fatma : Hi Fatima,

mlflow-export-import is still not a recommended solution and has no support from Databricks. This is clarified by Databricks SA's themselves.

I have developed an another approach where I am capturing the statistics of a particular run which is needed to be migrated to higher env. (like pre-prod or prod) and then creating a mlflow run with those statistics instead of re-running the entire huge model. This is working perfectly and is within the logical support of Databricks.