topic Re: Is it possible to get Job Run ID of notebook run by dbutils.notbook.run? in Data Engineering

Is it possible to get Job Run ID of notebook run by dbutils.notbook.run?

hanspetter — Wed, 02 Aug 2017 07:26:46 GMT

When running a notebook using dbutils.notebook.run from a master-notebook, an url to that running notebook is printed, i.e.:

Notebook job #223150

Notebook job #223151

Are there any ways to capture that Job Run ID (#223150 or #223151)? We have 50 or so notebooks that runs in parallell, and if one of them fails it would be nice to see the actual run of the notebook without clicking every url to find the correct one.

Thanks 🙂

Re: Is it possible to get Job Run ID of notebook run by dbutils.notbook.run?

AugustoElesbão — Mon, 11 Dec 2017 13:44:01 GMT

@hanspetter you can get the info via the command context:

dbutils.notebook.getContext.currentRunId

Besides of that, the following methods (vals) are available in the context:

jobGroup: Option[String] - unique command identifier that is injected by the driver.
rootRunId: Option[RunId]
tags: Map[String, String] - attribution tags injected by the webapp

Re: Is it possible to get Job Run ID of notebook run by dbutils.notbook.run?

naman1994 — Mon, 09 Apr 2018 11:52:50 GMT

Is there any development here, as currentRunId doesn't seem to work(i'm working in python though)

Re: Is it possible to get Job Run ID of notebook run by dbutils.notbook.run?

AugustoElesbão — Mon, 09 Apr 2018 14:14:37 GMT

@naman1994 check which Databricks Runtime version you're using. I know that this works with 3.5, but didn't test with 4.0 yet.

Re: Is it possible to get Job Run ID of notebook run by dbutils.notbook.run?

naman1994 — Mon, 09 Apr 2018 19:05:10 GMT

mine's 3.5 LTS, and it says, "NotebookHandler instance has no attribute 'getContext' "

Re: Is it possible to get Job Run ID of notebook run by dbutils.notbook.run?

naman1994 — Mon, 09 Apr 2018 19:05:33 GMT

It 'll be helpful if you could tell where you foundcurrentrunid, jobGroup, rootrunid etc, cz i cant seem to find these...

Anyways Thanks

Re: Is it possible to get Job Run ID of notebook run by dbutils.notbook.run?

AugustoElesbão — Tue, 10 Apr 2018 12:48:52 GMT

This code works in Scala only, there is no implementation for it in the Python runtime. And the information I got from a Databricks employee via email, those are internal APIs and they may change without notice.

Re: Is it possible to get Job Run ID of notebook run by dbutils.notbook.run?

naman1994 — Tue, 10 Apr 2018 12:50:42 GMT

Okay, thank you 🙂

Re: Is it possible to get Job Run ID of notebook run by dbutils.notbook.run?

naman1994 — Tue, 10 Apr 2018 13:32:06 GMT

In scala, have you ever returned this job run id to another notebook, since im calling many notebooks and i want the job run id returned from the respective notebooks to master. I think I could do it by returning an array, But im not sure if that is possible, cz If scala is like C++, then I wont be able to at all..

Anyways, i would appreciate your thoughts over this.

Thanks

Re: Is it possible to get Job Run ID of notebook run by dbutils.notbook.run?

Anonymous — Tue, 08 Jan 2019 15:49:11 GMT

FYI - for a Python notebook, you can get the run id in Scala and pass it in via the temp table:

Cell N:%scala val runId = dbutils.notebook.getContext .currentRunId .getOrElse(System.currentTimeMillis() / 1000L) .toString

Seq(runId).toDF("run_id").createOrReplaceTempView("run_id")Cell N + 1:

runId = spark.table("run_id").head()["run_id"]

Re: Is it possible to get Job Run ID of notebook run by dbutils.notbook.run?

staydol — Mon, 27 May 2019 15:21:53 GMT

In Python you should write smth like:

dbutils.notebook.entry_point.getDbutils().notebook().getContext().currentRunId().toString()

Re: Is it possible to get Job Run ID of notebook run by dbutils.notbook.run?

ShaunRyan1 — Wed, 03 Jul 2019 16:10:58 GMT

Definitely needs to be a Feature request. The Run ID is good, we know it's a notebook and therefore it should be the Notebook name not just the word "Notebook"

Also this dbutils.notebook.getContext.currentRunId

Is yielding a different number, when print out on success!

dbutils.notebook.exit(s"${notebook} #${dbutils.notebook.getContext.currentRunId} : success")

Notebook job #7890 - giving
MyNotebookName #Some(RunId(8188)) : success

Re: Is it possible to get Job Run ID of notebook run by dbutils.notbook.run?

hanspetter — Thu, 19 Mar 2020 10:36:24 GMT

Hi! It's been a long time since I have looked at this problem, but I revisited it this week and found a solution. 🙂

This returns a json containing information about the notebook:

dbutils.notebook.entry_point.getDbutils().notebook().getContext().toJson()

If the notebook has been triggered by dbutils.notebook.run, we can find the tag "jobId" here. So we can return the jobid using dbutils.notebook.exit(job_id):

import json 
notebook_info = json.loads(dbutils.notebook.entry_point.getDbutils().notebook().getContext().toJson())
try: 
    #The tag jobId does not exists when the notebook is not triggered by dbutils.notebook.run(...) 
    jobId = notebook_info["tags"]["jobId"] 
except: 
    jobId = -1 
dbutils.notebook.exit(jobId)

When executing the notebook we get:

jobid = dbutils.notebook.run(...) 
print(jobid)

This outputs:

Notebook job #1522478

1522478

Well, this only works when a notebook successfully completes. What we have done is that we are inserting the jobid and notebook path together with the current timestamp to a db-table and query the table when an exception occurs to get the jobid.

Re: Is it possible to get Job Run ID of notebook run by dbutils.notbook.run?

wahanand — Thu, 03 Sep 2020 02:07:40 GMT

Spark config value for "spark.databricks.clusterUsageTags.clusterAllTags" has Json which contains jobId,runId,clusterId... etc

The below article will help you.

https://community.databricks.com/s/feed/0D53f00001HKHjxCAH

Re: Is it possible to get Job Run ID of notebook run by dbutils.notbook.run?

ANILKUMARRAVADA — Tue, 11 May 2021 10:29:53 GMT

Well this has helped me a lot but i was unable to retrive browserhostname when i was running the notebook. while is was present documentation.. can you please through some light on this

Re: Is it possible to get Job Run ID of notebook run by dbutils.notbook.run?

DungTran — Tue, 10 Aug 2021 15:52:26 GMT

I would like to add that this works too:

dbutils.notebook.entry_point.getDbutils().notebook().getContext().jobId().get()

Re: Is it possible to get Job Run ID of notebook run by dbutils.notbook.run?

DungTran — Tue, 10 Aug 2021 15:53:05 GMT

this is a more simple version:

dbutils.notebook.entry_point.getDbutils().notebook().getContext().jobId().get()

Re: Is it possible to get Job Run ID of notebook run by dbutils.notbook.run?

xiangzhu — Tue, 05 Apr 2022 13:29:03 GMT

Thank you very mush for the share !

dbutils.notebook.entry_point is even not given by the official doc:

https://docs.databricks.com/dev-tools/databricks-utils.html#notebook-utility-dbutilsnotebook

Re: Is it possible to get Job Run ID of notebook run by dbutils.notbook.run?

415963 — Tue, 25 Jul 2023 15:18:30 GMT

Is there also a way to achieve this, when the notebook does not successfully complete. E.g.: when a structured streaming query fails.

Re: Is it possible to get Job Run ID of notebook run by dbutils.notbook.run?

Rodrigo_Mohr — Thu, 11 Apr 2024 19:49:47 GMT

I know this is an old thread, but sharing what is working for me well in Python now, for retrieving the run_id as well and building the entire link to that job run:

job_id = dbutils.notebook.entry_point.getDbutils().notebook().getContext().jobId().get()
run_id = dbutils.notebook.entry_point.getDbutils().notebook().getContext().currentRunId().get().id()
workspace_url = spark.conf.get("spark.databricks.workspaceUrl")
full_run_url = f"http://{workspace_url}/jobs/{job_id}/runs/{run_id}"