Automating the Migration of ML Models and Metadata to Unity Catalog

Himanshu4
Databricks Partner

‎10-07-2024 03:50 AM

Hello everyone,

I am currently working on a project where I need to migrate ML models from the Model Registry of one Databricks workspace to the Unity Catalog (schema/catalog) of another workspace. This migration needs to include not only the models themselves but also their associated metadata, such as model signatures, tags, and artifacts.

I would like to ask:

  1. Best Practices: What are the best practices for automating the migration process of ML models and their metadata between workspaces in Databricks?

  2. Automation Approaches: Are there specific tools, libraries, or scripts that can help automate this migration? Has anyone developed a custom solution that they could share?

  3. Handling Metadata: What specific metadata should I focus on preserving during the migration? How can I ensure that this metadata remains intact and accessible in the new workspace?

0 Kudos

gchandra
Databricks Employee
Databricks Employee

‎10-07-2024 05:22 AM

Deploy as Code is recommended.

https://learn.microsoft.com/en-us/azure/databricks/machine-learning/mlops/deployment-patterns#deploy...

If you need to copy the model as a Pickle file then use https://github.com/mlflow/mlflow-export-import

CICD implementation sample

https://github.com/gchandra10/ml_export_import_demo



~
0 Kudos