topic Re: Trained model artifact, CI/CD and Databricks without MLFlow. in Data Engineering

Trained model artifact, CI/CD and Databricks without MLFlow.

maranBH — Wed, 24 Nov 2021 14:21:55 GMT

Hi all,

We are constructing our CI/CD pipelines with the Repos feature following this guide:

https://databricks.com/blog/2021/09/20/part-1-implementing-ci-cd-on-databricks-using-databricks-notebooks-and-azure-devops.html

I'm trying to implement my pipes for models that haven't been trained with the MLFlow paradigm. The code promotion with repos is the same as explained in the link.

However, the model artifact, that is, the serialized file that contains the trained model

on MLFlow travels from its own infrastructure (the MLFlow artifact version control). It points either to the DB FileStore or any other storage previously set.

WITHOUT the MLFlow scheme: what are good practices for promoting the Model Artifact ? Here are my thoughts:

Save it on the data lake, and read the same direction throughout environments. Obstacle: How to add version control here??
When the release pipeline triggers, include it on the Repository artifact.
Any other?

I'm really confused about the solution.

Re: Trained model artifact, CI/CD and Databricks without MLFlow.

maranBH — Wed, 01 Dec 2021 11:42:01 GMT

Hi, thanks @Kaniz Fatma, is there any advice you can give me for this particular problem?

Re: Trained model artifact, CI/CD and Databricks without MLFlow.

Anonymous — Thu, 09 Dec 2021 22:19:55 GMT

@Rodrigo Maranzana - Thank you for your patience. I am looking for someone to help you.

Re: Trained model artifact, CI/CD and Databricks without MLFlow.

sean_owen — Thu, 06 Jan 2022 03:14:39 GMT

So you are managing your models with MLflow, and want to include them in a git repository?

You can do that in a CI/CD process; it would run the mlflow CLI to copy the model you want (e.g. model:/my_model/production) to a git checkout and then commit it, perhaps.

You can also treat MLflow as the 'source control' at runtime and have the production jobs refer to and access the latest model from MLflow directly.

Maybe I don't understand what you're trying to achieve here and you can clarify.