cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

How to version your workflows/jobs

ashdam
New Contributor III

WE would like to version control workflows/jobs over git, not the underlying notebooks but the job logic itself

is it possible?

2 ACCEPTED SOLUTIONS

Accepted Solutions

karthik_p
Esteemed Contributor

@ashdam NO cost is Involved, currently  it is in public preview, soon we can expect into GA

View solution in original post

daniel_sahal
Esteemed Contributor

@ashdam 
You've got three ways of doing the workflows deployment:

1. DAB (Databricks Asset Bundles)
2. Terraform (https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/job)
3. Use CLI/Python SDK through your notebook, ex. 

import requests

job_json = """
# Put JSON with job contents here
"""

api_token = nb_context['extraContext']['api_token']
ws_host = f"https://{nb_context['tags']['browserHostName']}"
job_api_url = f"{ws_host}/api/2.1/jobs/create"

auth_headers = {"Authorization" : f"Bearer {api_token}"}

create_job = requests.post(job_api_url, headers=auth_headers, data=job_json)

print("Status Code", create_job.status_code)
print("JSON Response ", create_job.json())

View solution in original post

9 REPLIES 9

ashdam
New Contributor III

where do you see the job's code inside a notebook?

ashdam_0-1698401582835.png

 

ashdam
New Contributor III

Another question if I want to make some changes at job level in dev but maintain the production job, whats the prodedure? Clone the production job to dev a folder make changes and then substitute?

And last question, if i want to run the job master for instance but executing in dev branch? is that possible?

karthik_p
Esteemed Contributor

@ashdam you can create multiple branches in git like dev, qa, main . dev/qa you can perform changes what ever you want to make. if you want those changes in main which is production branch then you can create PR and you can merge else you can just have main branch without any changes

ashdam
New Contributor III

@karthik_p yeah thats how git works but How do I put a job's code into GIT? (Not the notebook, the job's code)

How Do I test/parametrice the same job for working in dev/uaT/prod?

karthik_p
Esteemed Contributor

@ashdam dis you tried DAB in Databricks, it has version control mechanism for jobs to , try to leverage that functionality 

ashdam
New Contributor III

do you know if this has addtional cost?

karthik_p
Esteemed Contributor

@ashdam NO cost is Involved, currently  it is in public preview, soon we can expect into GA

daniel_sahal
Esteemed Contributor

@ashdam 
You've got three ways of doing the workflows deployment:

1. DAB (Databricks Asset Bundles)
2. Terraform (https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/job)
3. Use CLI/Python SDK through your notebook, ex. 

import requests

job_json = """
# Put JSON with job contents here
"""

api_token = nb_context['extraContext']['api_token']
ws_host = f"https://{nb_context['tags']['browserHostName']}"
job_api_url = f"{ws_host}/api/2.1/jobs/create"

auth_headers = {"Authorization" : f"Bearer {api_token}"}

create_job = requests.post(job_api_url, headers=auth_headers, data=job_json)

print("Status Code", create_job.status_code)
print("JSON Response ", create_job.json())

ashdam
New Contributor III

Thank you very much for all your answers

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group