topic Re: Retrieve job id and run id from scala in Data Engineering

Retrieve job id and run id from scala

Sunny — Wed, 08 Jun 2022 17:55:55 GMT

I need to retrieve job id and run id of the job from a jar file in Scala.

When I try to compile below code in IntelliJ, below error is shown.

import com.databricks.dbutils_v1.DBUtilsHolder.dbutils
 
object MainSNL {
 
  @throws(classOf[Exception])
  def main(args: Array[String]): Unit = {
 
  dbutils.notebook.getContext.tags("jobId").toString()
   dbutils.notebook.getContext.tags("runId").toString()
 
}
 
}

error: Symbol 'type com.databricks.backend.common.rpc.CommandContext' is missing from the classpath.

[ERROR] This symbol is required by 'method com.databricks.dbutils_v1.NotebookUtils.getContext'.

[ERROR] Make sure that type CommandContext is in your classpath and check for conflicting dependencies with `-Ylog-classpath`.

[ERROR] A full rebuild may help if 'NotebookUtils.class' was compiled against an incompatible version of com.databricks.backend.common.rpc.

[ERROR] dbutils.notebook.getContext.tags("jobId").toString()

[ERROR] ^

[ERROR] one error found

Re: Retrieve job id and run id from scala

Sunny — Thu, 09 Jun 2022 10:45:51 GMT

This is the same code I've used but receiving compilation error in IntelliJ.

Re: Retrieve job id and run id from scala

-werners- — Thu, 09 Jun 2022 13:17:21 GMT

Do you use databricks-connect with Intellij?

Re: Retrieve job id and run id from scala

Sunny — Thu, 09 Jun 2022 14:34:50 GMT

Re: Retrieve job id and run id from scala

Sunny — Fri, 17 Jun 2022 12:09:18 GMT

Hi @Kaniz Fatma Are there any alternatives to this, please advise

Re: Retrieve job id and run id from scala

ron_defreitas — Wed, 29 Jun 2022 18:22:20 GMT

Another method of getting access to this data is through the spark context, and this will not require use of the dbutils library.

If you check the environment tab of the spark cluster UI, you'll see that Databricks adds a number of properties to the spark config that can be easily retrieved using the native spark apis.

An example of the value of spark.databricks.clusterUsageTags.clusterName is "job-12345678901-run-987654-default" and you can retrieve this using spark.conf.get()

Re: Retrieve job id and run id from scala

Sunny — Thu, 30 Jun 2022 12:03:11 GMT

This is resolved by passing {{job_id}} and {{run_id}} as parameters to the jar

Re: Retrieve job id and run id from scala

Mohit_m — Tue, 05 Jul 2022 10:08:22 GMT

Maybe its worth going through the Task Parameter variables section of the below doc

https://docs.databricks.com/data-engineering/jobs/jobs.html#task-parameter-variables