topic Identify a job name in case notebook is triggered from another notebook/ in Data Engineering

Identify a job name in case notebook is triggered from another notebook/

Suman-Sourav — Tue, 17 Dec 2024 01:49:59 GMT

I am running an another notebook from one of the notebook as below. The main notebook is scheuled with a job/workflow.

dbutils.notebook.run("./ABC/XYZ/another_notebook", 1000)

Usually to get the workflow/job name i use any of the below 2 options.

1. desc history dbw_lakehouse.uat.table_name (if notebook loads to table)

2. Through API call

https://adb-URL.2.azuredatabricks.net/api/2.0/jobs/runs/get?run_id=103532871384632
https://adb-URL .2.azuredatabricks.net/api/2.0/jobs/get?job_id=1051355057291376

But in my scenario where i am calling notebook from extreanl notebook, i am not seeing the job name in any of the option output.

Auy suggesiton?

Re: Identify a job name in case notebook is triggered from another notebook/

Alberto_Umana — Tue, 17 Dec 2024 02:16:01 GMT

When you run a notebook from another notebook using dbutils.notebook.run, the job name of the parent notebook (the one that is scheduled with a job/workflow) is not directly accessible in the child notebook (the one being called). This is why you are not seeing the job name in the output of your usual methods

To work around this, you can pass the job name as a parameter to the child notebook. Here’s how you can do it:

Retrieve the job name in the parent notebook: You can use the Databricks REST API to get the job name. For example

import requests

import json

run_id = dbutils.notebook.entry_point.getDbutils().notebook().getContext().currentRunId().get()

response = requests.get(f"https://<databricks-instance>/api/2.0/jobs/runs/get?run_id={run_id}",

headers={"Authorization": f"Bearer {dbutils.notebook.entry_point.getDbutils().notebook().getContext().apiToken().get()}"})

job_name = response.json().get("job_name")

Pass the job name to the child notebook: When calling the child notebook, pass the job name as an argument:

dbutils.notebook.run("./ABC/XYZ/another_notebook", 1000, {"job_name": job_name})

Access the job name in the child notebook: In the child notebook, retrieve the job name from the arguments:

job_name = dbutils.widgets.get("job_name")

print(f"Job Name: {job_name}")

Re: Identify a job name in case notebook is triggered from another notebook/

Suman-Sourav — Tue, 17 Dec 2024 03:13:59 GMT

Thanks Alberto. I am aware of that solution but trying to get an option to get job name in the scenario i gave.

So i assume there is no way to get the job name in that scenario.