topic Re: I have a notebook in workspace, how to know in which job this particular notebook is referenced. in Data Engineering

I have a notebook in workspace, how to know in which job this particular notebook is referenced.

SS_RATH — Thu, 04 Apr 2024 03:35:50 GMT

I have a notebook in workspace, how to know in which job this particular notebook is referenced.

Re: I have a notebook in workspace, how to know in which job this particular notebook is referenced.

Ayushi_Suthar — Thu, 04 Apr 2024 05:36:14 GMT

Hi @SS_RATH, Good Day!

I want to inform you that there isn't a direct way to search for a job by the notebook it references. You will have to manually check each job to see which one is using the specific notebook you are interested in.

You can follow these steps to find out the job which refers a specific notebook:

1. Navigate to the Databricks workspace.
2. In the sidebar, click on 'Workflows'.
3. In the 'Jobs' tab, you will see a list of all the jobs.
4. Click on the name of each job to view its details.
5. In the 'Job UI page', look for the 'Notebook' section under 'Task options'. This section will show the path of the notebook that is being used by the job.

https://docs.databricks.com/en/workflows/jobs/create-run-jobs.html

Or you can use the Job API to list out all the Jobs and it will give you the complete details about the Jobs present in your workspace. Please refer to this document for the same: https://docs.databricks.com/api/workspace/jobs/list

Please let me know if this helps and leave a like if this information is useful, followups are appreciated.
Kudos
Ayushi

Re: I have a notebook in workspace, how to know in which job this particular notebook is referenced.

TamD — Mon, 14 Oct 2024 03:41:29 GMT

Hi @Ayushi_Suthar I don't see "Task options" in the Job UI. If I click on a Task, I see Task name, Type, Job, Depends on, Job parameters, Notifications, and Duration threshold. @SS_RATH , did you find an answer?

Re: I have a notebook in workspace, how to know in which job this particular notebook is referenced.

Panda — Mon, 14 Oct 2024 13:32:04 GMT

@SS_RATH @TamD There are couple of ways

Call Databricks REST API - Use the /api/2.1/jobs/list API to list and search through all jobs. Example: -

import requests workspace_url = "https://<databricks-instance>" databricks_token = "<your-databricks-token>" url = f"{workspace_url}/api/2.1/jobs/list" headers = {"Authorization": f"Bearer {databricks_token}"} response = requests.get(url, headers=headers) jobs = response.json().get('jobs', []) notebook_path = "/Workspace/Users/xxx@domain.com/MyNotebook" jobs_used_notebook = [] for job in jobs: job_name = job.get('settings', {}).get('name', 'Unnamed Job') # Get job name for task in job.get('settings', {}).get('tasks', []): if task.get('notebook_task', {}).get('notebook_path') == notebook_path: jobs_used_notebook.append({'job_id': job['job_id'], 'job_name': job_name}) if jobs_used_notebook: for job_info in jobs_used_notebook: print(f"Job Name: {job_info['job_name']}, Job ID: {job_info['job_id']}") else: print(f"No jobs found used the notebook path: {notebook_path}.")

2. Go to Databricks Workspace UI - Job Search enter your full path of the notebook Then Click serach. After result populate make sure choose your type in your case it's "Job"