topic Re: Databricks Jobs and CICD in Data Engineering

Databricks Jobs and CICD

labromb — Wed, 02 Nov 2022 16:07:42 GMT

Hi,

We currently leverage Azure DevOps to source control our notebooks and use CICD to publish the notebooks to different environments and this works very well.

We do not have the same functionality available for Databricks jobs (the ability to source control or deploy through CICD). Could we call the Databricks Jobs create API to promote the job definitions through environments or is there a better way of doing this?

Look forward to hearing from you

Re: Databricks Jobs and CICD

bergmaal — Tue, 22 Nov 2022 14:57:04 GMT

We're also missing this feature to have easy version control and CI/CD for Jobs/Workflows. The answer from @Kaniz Fatma does not cover the question as I see it.

Right now the only options we see is using Terraform, or the Jobs API 2.1 as superficially described in this post. But they both require some sort of custom setup. What we really need is a more integrated experience like we have for notebooks using Databricks repos.

It would be great to get some updates from Databricks Workflows team on this. Is this a feature on the roadmap? If not, it would be awesome to get some official advise on handling source control and CI/CD for Workflows/Jobs.

Re: Databricks Jobs and CICD

labromb — Wed, 23 Nov 2022 15:58:18 GMT

Hi @Kaniz Fatma interesting new feature for sure but not sure how this helps me with the challenge. As an aside, there probably also needs to be something similar for Databricks tables as the definition of the tables is essentially code.

Echoing @Anders Bergmål-Manneråk comments from yesterday....

Re: Databricks Jobs and CICD

saadansari-db — Wed, 14 Dec 2022 21:43:58 GMT

Hi @Brian Labrom and @Anders Bergmål-Manneråk - my team is looking at creating a set of features to provide simpler source control and CI/CD. Can you please send me an email at saad dot ansari at databricks dot com?

I would love to get more feedback incorporated into our approach here.

Re: Databricks Jobs and CICD

bergmaal — Thu, 15 Dec 2022 09:36:14 GMT

@Saad Ansari popped you an email now.

Re: Databricks Jobs and CICD

labromb — Thu, 15 Dec 2022 10:00:18 GMT

@Saad Ansari have done the same

Re: Databricks Jobs and CICD

JRT5933 — Thu, 26 Jan 2023 20:39:30 GMT

My team is currently looking at establishing REPO(s) for source control to start. I know I've seen some documentation for when a MERGE is completed to auto update the main branch in DBX remote repo. Does annyone have a template and/or best practices here?

Re: Databricks Jobs and CICD

sandeepmankikar — Wed, 26 Feb 2025 00:19:30 GMT

To manage Databricks jobs within a DevOps pipeline, start by exporting the job configuration as a JSON file from the Databricks workspace. Parameterize this JSON by replacing environment-specific values with placeholders. Integrate the parameterized JSON into your DevOps repository and use CI/CD tools to replace these placeholders with actual values during pipeline execution. Create the job using the Databricks CLI command databricks jobs create --json-file path_to_your_json_file.json. Additionally, set up a separate pipeline to delete jobs using the command databricks jobs delete --job-id .