topic Re: How to version your workflows/jobs in Data Engineering

How to version your workflows/jobs

ashdam — Fri, 27 Oct 2023 07:34:57 GMT

WE would like to version control workflows/jobs over git, not the underlying notebooks but the job logic itself

is it possible?

Re: How to version your workflows/jobs

ashdam — Fri, 27 Oct 2023 10:14:26 GMT

where do you see the job's code inside a notebook?

Re: How to version your workflows/jobs

ashdam — Fri, 27 Oct 2023 10:23:39 GMT

Another question if I want to make some changes at job level in dev but maintain the production job, whats the prodedure? Clone the production job to dev a folder make changes and then substitute?

And last question, if i want to run the job master for instance but executing in dev branch? is that possible?

Re: How to version your workflows/jobs

karthik_p — Fri, 27 Oct 2023 11:08:56 GMT

@ashdam you can create multiple branches in git like dev, qa, main . dev/qa you can perform changes what ever you want to make. if you want those changes in main which is production branch then you can create PR and you can merge else you can just have main branch without any changes

Re: How to version your workflows/jobs

ashdam — Fri, 27 Oct 2023 12:31:09 GMT

@karthik_p yeah thats how git works but How do I put a job's code into GIT? (Not the notebook, the job's code)

How Do I test/parametrice the same job for working in dev/uaT/prod?

Re: How to version your workflows/jobs

karthik_p — Sat, 28 Oct 2023 21:48:34 GMT

@ashdam dis you tried DAB in Databricks, it has version control mechanism for jobs to , try to leverage that functionality

Re: How to version your workflows/jobs

ashdam — Mon, 30 Oct 2023 09:36:27 GMT

do you know if this has addtional cost?

Re: How to version your workflows/jobs

karthik_p — Mon, 30 Oct 2023 14:43:47 GMT

@ashdam NO cost is Involved, currently it is in public preview, soon we can expect into GA

Re: How to version your workflows/jobs

daniel_sahal — Tue, 31 Oct 2023 05:45:28 GMT

@ashdam
You've got three ways of doing the workflows deployment:

1. DAB (Databricks Asset Bundles)
2. Terraform (https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/job)
3. Use CLI/Python SDK through your notebook, ex.

import requests job_json = """ # Put JSON with job contents here """ api_token = nb_context['extraContext']['api_token'] ws_host = f"https://{nb_context['tags']['browserHostName']}" job_api_url = f"{ws_host}/api/2.1/jobs/create" auth_headers = {"Authorization" : f"Bearer {api_token}"} create_job = requests.post(job_api_url, headers=auth_headers, data=job_json) print("Status Code", create_job.status_code) print("JSON Response ", create_job.json())

Re: How to version your workflows/jobs

ashdam — Tue, 31 Oct 2023 07:44:51 GMT

Thank you very much for all your answers