DBR 11.3 LTS - Specify spark.app.id for python notebook task

nttoan26 — Wed, 05 Jul 2023 10:24:37 GMT

Hi, I use Terraform databricks_job to submit a job with many tasks. IFAIK, each task is a separated Spark Application with a specific applicationId (https://spark.apache.org/docs/3.1.3/api/python/reference/api/pyspark.SparkContext.applicationId.html). These tasks are working fine except they have the same applicationId because Databricks uses app_yyyyMMddHHmmss as application id and all these apps are submitted at the same time by Terraform, hence they all have the same application. Does Databricks support specifying `spark.app.id` for python notebook task.

topic DBR 11.3 LTS - Specify spark.app.id for python notebook task in Get Started Discussions

DBR 11.3 LTS - Specify spark.app.id for python notebook task