โ08-22-2023 04:46 PM - edited โ08-22-2023 04:55 PM
Hi Databricks community,
I'm using Databricks Jobs Cluster to run some jobs. I'm setting the worker and driver type to AWS m6gd.large, which has 2 cores and 8G of memory each.
After seeing it's defaulting executor memory to 2G, I wanted to increase it, setting "spark.executor.memory 6g" on spark config on cluster setup. Upon setting it, it says I can't set it to such number, indicating the max value I can do is 2G (see the attachment). Given the worker has 8G memory, why is it limited to only 2G ? Similar situation for large worker types, the limit seems to be much lower than what should be available.
โ08-28-2023 12:07 PM - edited โ08-28-2023 12:08 PM
I think I found the right answer here: https://kb.databricks.com/en_US/clusters/spark-shows-less-memory
It seems it sets fixed size of ~4GB is used for internal node services. So depending on the node type, `spark.executor.memory` is fixed by Databricks and can't be adjusted further.
All the parameters mentioned above would be applicable for the leftover (2GB) available for the execution, as in the proportion within the leftover can be played around.
The thread helped me understand how the memory is being set. Good lesson.
โ08-25-2023 11:09 AM - edited โ08-25-2023 12:26 PM
@Retired_mod thanks for the reply.
I'm going to try it but I don't think it fully addresses the issue. According to your explanation, given a 8GB worker, on default, it will reserve ~800MB for the overhead memory. It still leaves ~7GB available, and yet, the platform limited spark.executor.memory to around 2GB (see screenshot in initial post). There could be other reserves for other things, but it is decent portion of memory left that should be available that I should be able to get allocation for.
EDIT: I tried lowering spark.executor.memoryOverheadFactor. The limitation value is still the same.
โ07-10-2024 08:17 AM
I do see this same issue with Photon enabled cluster. Mainly we are unable to change memory values as like with our own clusters.
โ08-28-2023 12:07 PM - edited โ08-28-2023 12:08 PM
I think I found the right answer here: https://kb.databricks.com/en_US/clusters/spark-shows-less-memory
It seems it sets fixed size of ~4GB is used for internal node services. So depending on the node type, `spark.executor.memory` is fixed by Databricks and can't be adjusted further.
All the parameters mentioned above would be applicable for the leftover (2GB) available for the execution, as in the proportion within the leftover can be played around.
The thread helped me understand how the memory is being set. Good lesson.
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group