Move whole workflow from Dev to Prod

[KKo]

I have a workflow created in Dev, now I want to move the whole thing to prod and schedule it. The workflow has multiple notebooks, dependent libraries, parameters and such. How to move the whole thing to prod, instead of moving each notebooks and recreate the workflow again in PROD? Or, what are the other options here? Thanks in advance!

[Aviral-Bhardwaj]

databricsk CLI will be helpful here, if the Prod is in Different Shard, if you don't know this then you have to do it manually by downloading DBC files to other shards.

[daniel_sahal]

@Kris Koirala​ 

You'll need CI/CD pipelines to do that programmatically. You can use Terraform/AzureARM/Bicep or any other tool that you (or you're planning to) use.

For example - Azure + Azure DevOps:

https://learn.microsoft.com/en-us/azure/databricks/dev-tools/ci-cd/ci-cd-azure-devops

[SRK]

Option 1:

You can use Terraform. with Azure Devops to automate the deployments:

https://www.databricks.com/blog/2022/12/5/databricks-workflows-through-terraform.html

Option 2:

You can use Databricks to automate the deployments:

Jobs CLI | Databricks on AWS

[Hubert-Dudek]

Alternatively, you can just click the three dots options in workflow and choose "view JSON" and save JSON. Then use it in the Rest API call to create new workflow/job using that JSON (but usually some part needs to be removed)