topic How can I deploy workflow jobs to another databricks workspace? in Data Engineering

How can I deploy workflow jobs to another databricks workspace?

Dean_Lovelace — Wed, 05 Apr 2023 10:22:49 GMT

I have created a number of workflows in the Databricks UI.

I now need to deploy them to a different workspace.

How can I do that?

Code can be deployed via Git, but the job definitions are stored in the workspace only.

Re: How can I deploy workflow jobs to another databricks workspace?

karthik_p — Wed, 05 Apr 2023 19:43:15 GMT

@Dean Lovelace If you are using Git setting config with git repo, you can use same repo in further environments and modify cluster related to new environment that you deployed , please check below article that should provide some info

https://docs.databricks.com/repos/ci-cd-techniques-with-repos.html

Re: How can I deploy workflow jobs to another databricks workspace?

Anonymous — Thu, 06 Apr 2023 06:33:08 GMT

Hi @Dean Lovelace

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!

Re: How can I deploy workflow jobs to another databricks workspace?

Dean_Lovelace — Thu, 06 Apr 2023 08:07:08 GMT

I can't see anything in this link related to moving jobs (schedules) between environments. I have git integration, but workflow jobs are not stored in git.

Re: How can I deploy workflow jobs to another databricks workspace?

Dean_Lovelace — Thu, 06 Apr 2023 08:24:28 GMT

The only way I can find to move workflow jobs (schedules) to another workspace is:-

1) Create a job in the databricks UI (Workflows -> Jobs -> Create Job)

2) Copy the json definition ("...", View JSON, Create, Copy)

3) Save the json locally or in the Git Repo

4) Create the job in a different workspace (amending the cluster id) with the Databricks CLI command "databricks jobs create --json-file ***.json".

Is this the only way?

Re: How can I deploy workflow jobs to another databricks workspace?

karthik_p — Thu, 06 Apr 2023 12:48:43 GMT

@Dean Lovelace Instead of all this, directly integrate your workflow with git and use same in further environments and update cluster.

Re: How can I deploy workflow jobs to another databricks workspace?

Dean_Lovelace — Thu, 06 Apr 2023 13:27:12 GMT

The workflows are 'integrated' with git, as in they get the code from a git branch each time they run. I am not seeing an option to store the actual workflow definitions in git (task name, script path, dependencies, retries, schedule etc). I have a workflow with multiple tasks, and the dependencies need to be fully tested prior to a production deployment.

Re: How can I deploy workflow jobs to another databricks workspace?

Mumrel — Fri, 05 May 2023 07:28:36 GMT

I have the same question:

Of course you can do everything with terraform, but that forces development from pipelines as code instead of a visual one. If you start out with terraform, each deployment is done the same way and straightforward, but is in code instead of visual editor. So this is a steep learning curve.

I think getting terraform code from an exiting pipeline would be great, but I am also not sure if that works. Maybe one could build the same an utilize the db cli, but this all seems a lot of effort for something that many projects need.

I have seen projects doing it for Azure Data Factory like this: Edit Pipelines in UI, get Json description, of pipeline, replace/insert environment parameters, deploy arm script to different environment. I am also looking for the most native Databricks way

Re: How can I deploy workflow jobs to another databricks workspace?

Mumrel — Fri, 05 May 2023 07:35:53 GMT

Here is a hint how one could custom build it, I guess.

Re: How can I deploy workflow jobs to another databricks workspace?

Radhikad — Fri, 09 Jun 2023 07:58:10 GMT

Hello everyone, I need the same help from databricks expert. I have created a 'Job1' job with runtime 12.2 in 'Datbricks1' workspace. I have integrated with Azure repo and tried deploying in 'ENV1' using CI/CD pipeline. It is successfully deployed in ENV1 but not on ENV2 of Databrick2 with same runtime 12.2. I mean, the jobs details are not reflecting in another workspace. Could anyone please please help.

Re: How can I deploy workflow jobs to another databricks workspace?

marct — Wed, 27 Mar 2024 08:59:50 GMT

Hi @Dean_Lovelace,

did you find a solution that you can share?

Thanks in advance

Re: How can I deploy workflow jobs to another databricks workspace?

cpradeep — Fri, 15 Nov 2024 14:45:45 GMT

@Dean_Lovelace

did you implement the solution ? Please share how you implemented CI/CD for workflow?

Re: How can I deploy workflow jobs to another databricks workspace?

itacdonev — Sun, 12 Jan 2025 10:29:51 GMT

It is not a seamless deploy but it worked for a one-shot transfer.

Here is what worked for me:

go to the original workflow and click "switch to code version (YAML)"
select all and copy
go to the new workspace and click create new job
click "switch to code version (YAML)" and click paste
in the new workspace go to the compute and select the compute you wish to use. Then clik on the View JSON and copy the compute ID
got back to the new job in the new workspace and paste this compute ID where ever you have "existing_cluster_id": XXXXXX (replace the XXX with the new id)
click save and view in visual mode
adjust as necessary

Re: How can I deploy workflow jobs to another databricks workspace?

Walter_C — Sun, 12 Jan 2025 12:47:07 GMT

@itacdonev great option provided, @Dean_Lovelace you can also select the option View JSON on the Workflow and move to the option create, with this code you can use the API https://docs.databricks.com/api/workspace/jobs/create and create the job in the desire workspace following the same job code including the same cluster configuration if using job cluster