cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Migration from self-managed MLflow to Databricks managed MLflow.

Maverick1
Valued Contributor II

Would it be possible to somehow save the data, metrics of all experiments captured by self-managed mlflow using A/mazon RDS, S3 as backend and then load it to databricks managed mlflow and make it available in the UI? This is required as a part of migration activity.

1 ACCEPTED SOLUTION

Accepted Solutions

Maverick1
Valued Contributor II

@Kaniz Fatmaโ€‹ : Hi Fatima,

mlflow-export-import is still not a recommended solution and has no support from Databricks. This is clarified by Databricks SA's themselves.

I have developed an another approach where I am capturing the statistics of a particular run which is needed to be migrated to higher env. (like pre-prod or prod) and then creating a mlflow run with those statistics instead of re-running the entire huge model. This is working perfectly and is within the logical support of Databricks.

View solution in original post

8 REPLIES 8

Kaniz_Fatma
Community Manager
Community Manager

Hi @Maverick! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers on the community have an answer to your question first. Or else I will follow up with my team and get back to you soon.Thanks.

User16783853898
Contributor

You can migrate your MLflow experiment and runs using the MLflow API with this toolkit: https://github.com/amesar/mlflow-export-import

@Andre Mesarovicโ€‹ : Is this solution, Databricks approved for enterprise usage?

I believe this is not an approved solution and also I can see many issues in it while doing import and export like nested metrics are not imported correctly etc.

User16783853898
Contributor

mlflow-export-import is an open source tool used by both OSS and Databricks users. It is in the process of being a recommended solution for Databricks MLflow migrations. Create a ticket for any issues at: https://github.com/amesar/mlflow-export-import/issues

Kaniz_Fatma
Community Manager
Community Manager

Hi @Saurabh Vermaโ€‹  , Just a friendly follow-up. Do you still need help or the above responses help you to find the solution? Please let us know.

Maverick1
Valued Contributor II

@Kaniz Fatmaโ€‹ : Hi Fatima,

mlflow-export-import is still not a recommended solution and has no support from Databricks. This is clarified by Databricks SA's themselves.

I have developed an another approach where I am capturing the statistics of a particular run which is needed to be migrated to higher env. (like pre-prod or prod) and then creating a mlflow run with those statistics instead of re-running the entire huge model. This is working perfectly and is within the logical support of Databricks.

Hi @Saurabh Vermaโ€‹ , I'm glad that you have got a solution to your question. Would you like to mark your answer as the best?

Hi @Saurabh Vermaโ€‹ , mlflow-export-import is in the process of being the Databricks recommended solution. We will have it in our official DB docs soon. It is based on the open source MLflow API and works for both OSS and DB. We have over 105 customers using it. It is the only way to migrate MLflow objects from one tracking server (workspace) to another.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group