Adding spark_conf tag on Jobs API

shan_chandra — Thu, 23 Feb 2023 20:31:12 GMT

using Jobs API, when we create a new job to run on an interactive cluster, can we add spark_conf tag and specify spark config tuning parameters?

Re: Adding spark_conf tag on Jobs API

shan_chandra — Thu, 23 Feb 2023 20:53:26 GMT

spark_conf needs to be set prior to the start of the cluster or have to restart the existing cluster. Hence, the spark_conf tag is available only on the job_cluster. you may have to set the configs manually on the interactive cluster prior to using Jobs API. we cannot change interactive clusters from Jobs API only restart them if they have terminated.

topic Adding spark_conf tag on Jobs API in Data Engineering

Adding spark_conf tag on Jobs API

Re: Adding spark_conf tag on Jobs API