What is the best to way to not deploy/run a workflow in production?

sergiopolimante — Wed, 23 Oct 2024 22:55:30 GMT

I am building and MLOps architecture.
I do not want to deploy the training workflow to prod.

My first approach was to selectively not deploy the workflow to prod, but this does not seem to be possible as in this thread:
https://community.databricks.com/t5/data-engineering/databricks-bundles-how-to-select-which-jobs-resources-to-deploy/td-p/62023

Then, I thought of checking in the first task of training workflow if it is prod and then skipping the whole job at once, not just the first task. But it does not seem to be possible to skip a whole job, only specific tasks of it, as per this thread:
https://community.databricks.com/t5/data-engineering/gracefully-stop-a-job-based-on-condition/td-p/37610

The only approach left is to check all job tasks in the production environment and skip them, but it is not an elegant approach.

Is there a better approach?

Re: What is the best to way to not deploy/run a workflow in production?

Walter_C — Mon, 04 Nov 2024 20:37:40 GMT

Target Override Feature: You can use the target override feature to specify different configurations for different environments. However, this does not provide a direct way to exclude specific job resources.
Environment-Specific Folders: Another approach is to create environment-specific folders and include only the necessary resources for each environment. This requires maintaining separate configurations for each environment, which can be cumbersome.

topic What is the best to way to not deploy/run a workflow in production? in Machine Learning

What is the best to way to not deploy/run a workflow in production?

Re: What is the best to way to not deploy/run a workflow in production?