Executor memory increase limitation based on node type

[938452]

Hi Databricks community,

I'm using Databricks Jobs Cluster to run some jobs. I'm setting the worker and driver type to AWS m6gd.large, which has 2 cores and 8G of memory each.

After seeing it's defaulting executor memory to 2G, I wanted to increase it, setting "spark.executor.memory 6g" on spark config on cluster setup. Upon setting it, it says I can't set it to such number, indicating the max value I can do is 2G (see the attachment). Given the worker has 8G memory, why is it limited to only 2G ? Similar situation for large worker types, the limit seems to be much lower than what should be available.

[938452]

@Retired_mod thanks for the reply.

I'm going to try it but I don't think it fully addresses the issue. According to your explanation, given a 8GB worker, on default, it will reserve ~800MB for the overhead memory. It still leaves ~7GB available, and yet, the platform limited  spark.executor.memory to around 2GB (see screenshot in initial post). There could be other reserves for other things, but it is decent portion of memory left that should be available that I should be able to get allocation for.

EDIT: I tried lowering spark.executor.memoryOverheadFactor. The limitation value is still the same.

[2vinodhkumar]

I do see this same issue with Photon enabled cluster. Mainly we are unable to change memory values as like with  our own clusters.

[938452 · Accepted solution]

I think I found the right answer here: https://kb.databricks.com/en_US/clusters/spark-shows-less-memory

It seems it sets fixed size of ~4GB is used for internal node services. So depending on the node type, `spark.executor.memory` is fixed by Databricks and can't be adjusted further.

All the parameters mentioned above would be applicable for the leftover (2GB) available for the execution, as in the proportion within the leftover can be played around.

The thread helped me understand how the memory is being set. Good lesson.