topic Re: Cloning of Workflow from One env to different env using Job API in Data Engineering

Cloning of Workflow from One env to different env using Job API

ameet9257 — Wed, 20 Nov 2024 18:28:10 GMT

Hi Team,

One of my team members recently shared one requirement: he wants to migrate the 10 Workflows from the sandbox to the dev environment to run his model in dev env.

I wanted to move all these workflows in an automated way and one of the solutions that came to my mind is Databricks Job API.

Below are the quick steps that I have followed.

1. Get the existing workflow configuration from the Databricks UI.

Open existing workflow --> Click on View JSON

2. Choose the Create option and copy the JSON.

3. Call the JOB API and pass the copy JSON to the API body.

POST https://XXXXXX.azuredatabricks.net/api/2.1/jobs/create

Using these three steps, I migrated the workflow to a new environment.

I think the best way is to go through the terraform and CI-CD for production env but if there is no such thing in place I found this as the quickest option.

Please do share your solutions as well.

#Job API #databricks #workflows

Re: Cloning of Workflow from One env to different env using Job API

Stefan-Koch — Thu, 21 Nov 2024 05:17:05 GMT

Hi Amet

Databricks Asset Bundles are designed precisely for the requirements you have.

https://docs.databricks.com/en/dev-tools/bundles/index.html

You can also transfer existing jobs that were created manually into a bundle. This works like this, for example:

databricks bundle generate job --existing-job-id 6565621249

Re: Cloning of Workflow from One env to different env using Job API

ameet9257 — Thu, 21 Nov 2024 07:42:59 GMT

@Stefan-Koch

Thanks. This looks interesting and I will try this.