topic Re: How do I get the current cluster id? in Data Engineering

How do I get the current cluster id?

ammobear — Mon, 11 Feb 2019 23:43:52 GMT

I am adding Application Insights telemetry to my Databricks jobs and would like to include the cluster ID of the job run. How can I access the cluster id at run time?

The requirement is that my job can programmatically retrieve the cluster id to insert into all telemetry. Retrieving the cluster ID through the UI will not be sufficient.

I don't see any dbutils commands that would be of use.

Re: How do I get the current cluster id?

tonyp — Tue, 12 Feb 2019 04:31:44 GMT

In Databricks click on your cluster in the Clusters tab, Change the UI interface to json, It will give the all details about your cluster

Re: How do I get the current cluster id?

ammobear — Tue, 12 Feb 2019 16:33:18 GMT

Thank you for your answer. I have added more detail to my question. Unfortunately, the UI will not work as I need my job code to programmatically pull the cluster id.

Re: How do I get the current cluster id?

Arti — Tue, 12 Feb 2019 18:23:12 GMT

You can use the cluster node initiaization script to grab the environment variable DB_CLUSTER_ID. Refer here https://docs.databricks.com/user-guide/clusters/init-scripts.html#environment-variables. Also from the notebook, the following also works :

spark.conf.get("spark.databricks.clusterUsageTags.clusterName")

spark.conf.get("spark.databricks.clusterUsageTags.clusterId")

Re: How do I get the current cluster id?

ammobear — Tue, 12 Feb 2019 18:54:16 GMT

That's perfect! Thank you for your help Arti.

Do you know if the properties of

clusterUsageTabs

are documented anywhere? I wonder what other useful properties I might be able to log.

Re: How do I get the current cluster id?

Arti — Tue, 12 Feb 2019 19:00:33 GMT

I am not sure if these are documented. Use the following code to get the entire list:

%scala

val configMap = spark.conf.getAll

configMap.foreach(println)

Re: How do I get the current cluster id?

ammobear — Tue, 12 Feb 2019 19:31:39 GMT

Thanks! Feel free to make your first comment a separate post and I'll mark it as the answer.

Re: How do I get the current cluster id?

Arti — Wed, 13 Feb 2019 04:08:03 GMT

You can use the cluster node initiaization script to grab the environment variable DB_CLUSTER_ID. Refer here https://docs.databricks.com/user-guide/clusters/init-scripts.html#environment-variables.

Also from the notebook , the following also works :

spark.conf.get("spark.databricks.clusterUsageTags.clusterName")

spark.conf.get("spark.databricks.clusterUsageTags.clusterId")

The entire list of spark properties is available in UI in Spark UI --> Environment tab.

Hope this helps!

Re: How do I get the current cluster id?

tonyp — Wed, 13 Feb 2019 04:20:01 GMT

Thank you for your answer.@Arti

Re: How do I get the current cluster id?

EricBellet — Thu, 03 Oct 2019 16:47:22 GMT

Hi, I'm trying to install a library using the init script option. But the $DB_CLUSTER_ID it is empty, why? Also I tried to do in a Notebook cell a %sh echo $DB_CLUSTER_ID and also is empty

curl -X POST https://dbc-myid.cloud.databricks.com/api/2.0/libraries/install -H 'Cache-Control:no-cache' -H 'Content-Type:application/json' -H 'Authorization: Bearer mytoken' -d '{"cluster_id": "$DB_CLUSTER_ID", "libraries": [{"egg": "dbfs:/FileStore/jars/library-0.1-py3.7.egg"}]}'

Re: How do I get the current cluster id?

Arti — Mon, 07 Oct 2019 23:21:26 GMT

Are you using cluster init script or global init script ?

DB_CLUSTER_ID - This env variable is available only during the cluster init script execution. I guess you will not be able to get the values for the these init script's env variables if you try from notebook.

Re: How do I get the current cluster id?

EricBellet — Tue, 08 Oct 2019 16:06:35 GMT

I fixed, it should be "'$DB_CLUSTER_ID'"