Adding spark_conf tag on Jobs API

[shan_chandra]

using Jobs API, when we create a new job to run on an interactive cluster, can we add spark_conf tag and specify spark config tuning parameters?

[shan_chandra · Accepted solution]

 spark_conf needs to be set prior to the start of the cluster or have to restart the existing cluster. Hence, the spark_conf tag is available only on the job_cluster. you may have to set the configs manually on the interactive cluster prior to using Jobs API. we cannot change interactive clusters from Jobs API only restart them if they have terminated.