Thank you very mush for the share !

dbutils.notebook.entry_point is even not given by the official doc:

https://docs.databricks.com/dev-tools/databricks-utils.html#notebook-utility-dbutilsnotebook

Is there also a way to achieve this, when the notebook does not successfully complete. E.g.: when a structured streaming query fails.

Spark config value for "spark.databricks.clusterUsageTags.clusterAllTags" has Json which contains jobId,runId,clusterId... etc

The below article will help you.

https://community.databricks.com/s/feed/0D53f00001HKHjxCAH

I would like to add that this works too:

I know this is an old thread, but sharing what is working for me well in Python now, for retrieving the run_id as well and building the entire link to that job run:

job_id = dbutils.notebook.entry_point.getDbutils().notebook().getContext().jobId().get()
run_id = dbutils.notebook.entry_point.getDbutils().notebook().getContext().currentRunId().get().id()
workspace_url = spark.conf.get("spark.databricks.workspaceUrl")
full_run_url = f"http://{workspace_url}/jobs/{job_id}/runs/{run_id}"

Thanks for the response @Manoj5 - I had to use this "safeToJson()" option too because all of the previous suggestions in this thread were erroring out for me with a message like "py4j.security.Py4JSecurityException: Method public java.lang.String com.databricks.backend.common.rpc.CommandContext.toJson() is not whitelisted on class class com.databricks.backend.common.rpc.CommandContext"

Also, as a more general comment, it's incredibly frustrating that none of this is documented, or if it is, it's organized so poorly that I cannot find it.