Cloning of Workflow from One env to different env using Job API

ameet9257
Contributor

‎11-20-2024 10:28 AM

Hi Team,

One of my team members recently shared one requirement: he wants to migrate the 10 Workflows from the sandbox to the dev environment to run his model in dev env.

I wanted to move all these workflows in an automated way and one of the solutions that came to my mind is Databricks Job API.

Below are the quick steps that I have followed.

1. Get the existing workflow configuration from the Databricks UI. 

Open existing workflow --> Click on View JSON 

ameet9257_0-1732126423665.png

2. Choose the Create option and copy the JSON.

ameet9257_1-1732126508104.png

3. Call the JOB API and pass the copy JSON to the API body.

POST https://XXXXXX.azuredatabricks.net/api/2.1/jobs/create

ameet9257_2-1732126684331.png

Using these three steps, I migrated the workflow to a new environment. 

I think the best way is to go through the terraform and CI-CD for production env but if there is no such thing in place I found this as the quickest option. 

Please do share your solutions as well. 

 

#Job API #databricks #workflows

Amit Prajapati
Labels:
0 Kudos

Stefan-Koch
Databricks Partner

‎11-20-2024 09:17 PM

Hi Amet

Databricks Asset Bundles are designed precisely for the requirements you have.

https://docs.databricks.com/en/dev-tools/bundles/index.html

You can also transfer existing jobs that were created manually into a bundle. This works like this, for example: 

databricks bundle generate job --existing-job-id 6565621249

 

ameet9257
Contributor

‎11-20-2024 11:42 PM

@Stefan-Koch 

Thanks. This looks interesting and I will try this.

 

Amit Prajapati
0 Kudos