Spark config ignored in job run settings

[dermoritz]

i am talking about this setting:

i tried so far:

spark.executor.cores 8
and
spark.log.level INFO
Both documented here: https://spark.apache.org/docs/latest/configuration.html
but neither i see effect nor i see them set if i check sparkui -> environment tab. there the settings are not set at all.
here the details of an actual run:

Am i doing it wrong? Or is it reset/ ignored for some other reason?
So how to set specific spark settings - in environment, NOT in code? 

[emma_s]

Hi,

You're not putting in the wrong place, it's just that Databricks doesn't allow certain configs, because they are managed by Databricks for you. For example your core spark config you've shown above won't be recognised as this is set by selected compute type. So rather than specifying the number of cores in your spark config you would select the compute that had the desired number of cores. In a serverless scenario it should autoscale to the optimum number of cores. 

For the log.level setting, i've just tested with my own job and it does impact the run. You can view it in the spark ui for the job after it's run, I tested it by setting it to warn. And my environment setting shows as WARN.

I hope this helps.

Many Thanks,

Emma

[dermoritz]

Hi Emma,
thx. My problem regarding CPU is: i have a workload that need much memory per task and there seem to be no compute (on azure) with more than 8GB per Core.
But i found a solution that indeed works: "spark.task.cpus 2" (so the settings work in general - my assumption was wrong)
this doubles the available ram per task - but halfs CPU utilization. but this is fine for the moment.
Regarding the log setting: i assume that this setting is also set by again my expectation about result is wrong: in run "output" i see all errors and i want to see my logs there - i know guess that it will always only show error logs or print and never user log regardless the setting.

[emma_s]

HI,

Glad you were able to resolve it. I would reach out to your Microsoft account team if you don't have the compute you require. They can sometimes do things to make more available.

Thanks,

Emma

[emma_s]

Hi,

You're not putting in the wrong place, it's just that Databricks doesn't allow certain configs, because they are managed by Databricks for you. For example your core spark config you've shown above won't be recognised as this is set by selected compute type. So rather than specifying the number of cores in your spark config you would select the compute that had the desired number of cores. In a serverless scenario it should autoscale to the optimum number of cores. 

For the log.level setting, i've just tested with my own job and it does impact the run. You can view it in the spark ui for the job after it's run, I tested it by setting it to warn. And my environment setting shows as WARN.

I hope this helps.

Many Thanks,

Emma