04-05-2023 03:22 AM
I have created a number of workflows in the Databricks UI.
I now need to deploy them to a different workspace.
How can I do that?
Code can be deployed via Git, but the job definitions are stored in the workspace only.
04-05-2023 12:43 PM
@Dean Lovelace If you are using Git setting config with git repo, you can use same repo in further environments and modify cluster related to new environment that you deployed , please check below article that should provide some info
https://docs.databricks.com/repos/ci-cd-techniques-with-repos.html
04-06-2023 01:07 AM
I can't see anything in this link related to moving jobs (schedules) between environments. I have git integration, but workflow jobs are not stored in git.
04-05-2023 11:33 PM
Hi @Dean Lovelace
Thank you for posting your question in our community! We are happy to assist you.
To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?
This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!
04-06-2023 01:24 AM
The only way I can find to move workflow jobs (schedules) to another workspace is:-
1) Create a job in the databricks UI (Workflows -> Jobs -> Create Job)
2) Copy the json definition ("...", View JSON, Create, Copy)
3) Save the json locally or in the Git Repo
4) Create the job in a different workspace (amending the cluster id) with the Databricks CLI command "databricks jobs create --json-file ***.json".
Is this the only way?
04-06-2023 05:48 AM
@Dean Lovelace Instead of all this, directly integrate your workflow with git and use same in further environments and update cluster.
04-06-2023 06:27 AM
The workflows are 'integrated' with git, as in they get the code from a git branch each time they run. I am not seeing an option to store the actual workflow definitions in git (task name, script path, dependencies, retries, schedule etc). I have a workflow with multiple tasks, and the dependencies need to be fully tested prior to a production deployment.
03-27-2024 01:59 AM
05-05-2023 12:28 AM
I have the same question:
Of course you can do everything with terraform, but that forces development from pipelines as code instead of a visual one. If you start out with terraform, each deployment is done the same way and straightforward, but is in code instead of visual editor. So this is a steep learning curve.
I think getting terraform code from an exiting pipeline would be great, but I am also not sure if that works. Maybe one could build the same an utilize the db cli, but this all seems a lot of effort for something that many projects need.
I have seen projects doing it for Azure Data Factory like this: Edit Pipelines in UI, get Json description, of pipeline, replace/insert environment parameters, deploy arm script to different environment. I am also looking for the most native Databricks way
05-05-2023 12:35 AM
06-09-2023 12:58 AM
Hello everyone, I need the same help from databricks expert. I have created a 'Job1' job with runtime 12.2 in 'Datbricks1' workspace. I have integrated with Azure repo and tried deploying in 'ENV1' using CI/CD pipeline. It is successfully deployed in ENV1 but not on ENV2 of Databrick2 with same runtime 12.2. I mean, the jobs details are not reflecting in another workspace. Could anyone please please help.
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group