How to version your workflows/jobs

[ashdam]

WE would like to version control workflows/jobs over git, not the underlying notebooks but the job logic itself

is it possible?

[Kaniz]

Hi @ashdam, Yes, it's possible to version control your workflows/jobs in Databricks using Git. When a job is created in Databricks, it is associated with a notebook or a set of notebooks. The code for the job is usually included in the notebook. However, you can separate the job logic from the notebook and store it in a separate file that can be version-controlled in Git.

[ashdam]

where do you see the job's code inside a notebook?

[ashdam]

Another question if I want to make some changes at job level in dev but maintain the production job, whats the prodedure? Clone the production job to dev a folder make changes and then substitute?

And last question, if i want to run the job master for instance but executing in dev branch? is that possible?

[karthik_p]

@ashdam you can create multiple branches in git like dev, qa, main . dev/qa you can perform changes what ever you want to make. if you want those changes in main which is production branch then you can create PR and you can merge else you can just have main branch without any changes

[ashdam]

@karthik_p yeah thats how git works but How do I put a job's code into GIT? (Not the notebook, the job's code)

How Do I test/parametrice the same job for working in dev/uaT/prod?

[karthik_p]

@ashdam dis you tried DAB in Databricks, it has version control mechanism for jobs to , try to leverage that functionality 

[ashdam]

do you know if this has addtional cost?

[karthik_p · Accepted solution]

@ashdam NO cost is Involved, currently  it is in public preview, soon we can expect into GA

[daniel_sahal · Accepted solution]

@ashdam 
You've got three ways of doing the workflows deployment:

1. DAB (Databricks Asset Bundles)
2. Terraform (https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/job)
3. Use CLI/Python SDK through your notebook, ex. 

import requests

job_json = """
# Put JSON with job contents here
"""

api_token = nb_context['extraContext']['api_token']
ws_host = f"https://{nb_context['tags']['browserHostName']}"
job_api_url = f"{ws_host}/api/2.1/jobs/create"

auth_headers = {"Authorization" : f"Bearer {api_token}"}

create_job = requests.post(job_api_url, headers=auth_headers, data=job_json)

print("Status Code", create_job.status_code)
print("JSON Response ", create_job.json())

[ashdam]

Thank you very much for all your answers