- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-27-2023 12:34 AM
WE would like to version control workflows/jobs over git, not the underlying notebooks but the job logic itself
is it possible?
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-30-2023 07:43 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-30-2023 10:45 PM
@ashdam
You've got three ways of doing the workflows deployment:
1. DAB (Databricks Asset Bundles)
2. Terraform (https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/job)
3. Use CLI/Python SDK through your notebook, ex.
import requests
job_json = """
# Put JSON with job contents here
"""
api_token = nb_context['extraContext']['api_token']
ws_host = f"https://{nb_context['tags']['browserHostName']}"
job_api_url = f"{ws_host}/api/2.1/jobs/create"
auth_headers = {"Authorization" : f"Bearer {api_token}"}
create_job = requests.post(job_api_url, headers=auth_headers, data=job_json)
print("Status Code", create_job.status_code)
print("JSON Response ", create_job.json())
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-27-2023 03:14 AM
where do you see the job's code inside a notebook?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-27-2023 03:23 AM
Another question if I want to make some changes at job level in dev but maintain the production job, whats the prodedure? Clone the production job to dev a folder make changes and then substitute?
And last question, if i want to run the job master for instance but executing in dev branch? is that possible?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-27-2023 04:08 AM
@ashdam you can create multiple branches in git like dev, qa, main . dev/qa you can perform changes what ever you want to make. if you want those changes in main which is production branch then you can create PR and you can merge else you can just have main branch without any changes
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-27-2023 05:31 AM
@karthik_p yeah thats how git works but How do I put a job's code into GIT? (Not the notebook, the job's code)
How Do I test/parametrice the same job for working in dev/uaT/prod?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-28-2023 02:48 PM
@ashdam dis you tried DAB in Databricks, it has version control mechanism for jobs to , try to leverage that functionality
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-30-2023 02:36 AM
do you know if this has addtional cost?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-30-2023 07:43 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-30-2023 10:45 PM
@ashdam
You've got three ways of doing the workflows deployment:
1. DAB (Databricks Asset Bundles)
2. Terraform (https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/job)
3. Use CLI/Python SDK through your notebook, ex.
import requests
job_json = """
# Put JSON with job contents here
"""
api_token = nb_context['extraContext']['api_token']
ws_host = f"https://{nb_context['tags']['browserHostName']}"
job_api_url = f"{ws_host}/api/2.1/jobs/create"
auth_headers = {"Authorization" : f"Bearer {api_token}"}
create_job = requests.post(job_api_url, headers=auth_headers, data=job_json)
print("Status Code", create_job.status_code)
print("JSON Response ", create_job.json())
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-31-2023 12:44 AM
Thank you very much for all your answers