topic Re: How can I customize the Notebook Job # while using dbutils.notebook.run method? in Data Engineering

How can I customize the Notebook Job # while using dbutils.notebook.run method?

soumyaPattnaik — Tue, 01 Nov 2022 14:34:26 GMT

When running multiple notebooks parallelly using dbutils.notebook.run from a parent notebook, an url to that running notebook is printed, like below

Notebook job #211371132480519

Is there a way I can print the notebook name or some customized string instead of this job id. I am running multiple notebooks (more than 100 at a time) - so with just the job id it is difficult to identify which notebook run is mapped with which job id

Thanks in advance!

Re: How can I customize the Notebook Job # while using dbutils.notebook.run method?

Debayan — Thu, 03 Nov 2022 07:15:52 GMT

Hi @Soumya Pattnaik , Configurations for the job can be updated using the reset or update endpoint in jobs API 2.1 (https://docs.databricks.com/dev-tools/api/latest/jobs.html).

Also, you can use cluster tags, which allow you to easily monitor the cost of cloud resources used by various groups in your organization. You can specify tags as key-value pairs when you create a cluster, and Azure Databricks applies these tags to cloud resources like VMs and disk volumes etc.

You can monitor usage using cluster or pool tags. (https://docs.databricks.com/administration-guide/account-settings/usage-detail-tags-aws.html).

For convenience, Azure Databricks applies four default tags to each cluster:

Vendor

Creator

ClusterName

and

ClusterId.

Re: How can I customize the Notebook Job # while using dbutils.notebook.run method?

soumyaPattnaik — Thu, 29 Jun 2023 08:17:46 GMT

Hi @Debayan Thank you for your reply.
However, the answer I am looking for is : how to print/get a more meaningful name of the jobs when running multiple notebooks parallelly using dbutils.notebook.run from a parent notebook.

Now in the parent notebook console something like this appears: (in case I am starting 2 notebook run from the parent NB)

Notebook job #211371132480519
Notebook job #2113711324805120

I am expecting something like below:

<path_to_nb1> or name of notebook1
<path_to_nb2> or name of notebook2

Re: How can I customize the Notebook Job # while using dbutils.notebook.run method?

ilarsen — Tue, 26 Sep 2023 20:48:36 GMT

+1, I'd like to know this as well.

Guess we'll just have to play around and see what we can do...