I have a notebook in workspace, how to know in which job this particular notebook is referenced.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-03-2024 08:35 PM
I have a notebook in workspace, how to know in which job this particular notebook is referenced.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-03-2024 10:36 PM
Hi @SS_RATH, Good Day!
I want to inform you that there isn't a direct way to search for a job by the notebook it references. You will have to manually check each job to see which one is using the specific notebook you are interested in.
You can follow these steps to find out the job which refers a specific notebook:
1. Navigate to the Databricks workspace.
2. In the sidebar, click on 'Workflows'.
3. In the 'Jobs' tab, you will see a list of all the jobs.
4. Click on the name of each job to view its details.
5. In the 'Job UI page', look for the 'Notebook' section under 'Task options'. This section will show the path of the notebook that is being used by the job.
https://docs.databricks.com/en/workflows/jobs/create-run-jobs.html
Or you can use the Job API to list out all the Jobs and it will give you the complete details about the Jobs present in your workspace. Please refer to this document for the same: https://docs.databricks.com/api/workspace/jobs/list
Please let me know if this helps and leave a like if this information is useful, followups are appreciated.
Kudos
Ayushi
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-13-2024 08:40 PM - edited 10-13-2024 08:41 PM
Hi @Ayushi_Suthar I don't see "Task options" in the Job UI. If I click on a Task, I see Task name, Type, Job, Depends on, Job parameters, Notifications, and Duration threshold. @SS_RATH , did you find an answer?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-14-2024 06:30 AM - edited 10-14-2024 06:32 AM
@SS_RATH @TamD There are couple of ways
- Call Databricks REST API - Use the /api/2.1/jobs/list API to list and search through all jobs. Example: -
import requests
workspace_url = "https://<databricks-instance>"
databricks_token = "<your-databricks-token>"
url = f"{workspace_url}/api/2.1/jobs/list"
headers = {"Authorization": f"Bearer {databricks_token}"}
response = requests.get(url, headers=headers)
jobs = response.json().get('jobs', [])
notebook_path = "/Workspace/Users/xxx@domain.com/MyNotebook"
jobs_used_notebook = []
for job in jobs:
job_name = job.get('settings', {}).get('name', 'Unnamed Job') # Get job name
for task in job.get('settings', {}).get('tasks', []):
if task.get('notebook_task', {}).get('notebook_path') == notebook_path:
jobs_used_notebook.append({'job_id': job['job_id'], 'job_name': job_name})
if jobs_used_notebook:
for job_info in jobs_used_notebook:
print(f"Job Name: {job_info['job_name']}, Job ID: {job_info['job_id']}")
else:
print(f"No jobs found used the notebook path: {notebook_path}.")
2. Go to Databricks Workspace UI - Job Search enter your full path of the notebook Then Click serach. After result populate make sure choose your type in your case it's "Job"