cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Driver: how much memory is actually available?

dbuserng
New Contributor

I have a cluster where Driver type is Standard_DS3_v2 (14GB Memory and 4 Cores). When I use free -h command in Web terminal (see attached screenshot) I get the response that I only have 8.9GB memory available on my driver - why is that?

fyi, spark.driver.memory is not set

 

1 REPLY 1

Sidhant07
Databricks Employee
Databricks Employee

Hi @dbuserng ,

The free -h command in the web terminal shows only 8.9GB of available memory on your driver, which is a Standard_DS3_v2 instance with 14GB of memory, because Databricks has services running on each node. This means the maximum allowable memory for Spark is less than the total memory capacity of the VM reported by the cloud provider. Even if spark.driver.memory is not explicitly set, some memory is reserved for these services, reducing the amount available for Spark processes.Screenshot 2025-01-30 at 1.45.35โ€ฏPM.pngScreenshot 2025-01-30 at 1.45.49โ€ฏPM.png

As a preface, the maximum memory size given to Spark is constrained by the other internal services we run on the VM. The current formula for this is

container_size = (vm_size * 0.97 - 4800MB)

and is implicitly defined in NodeTypeConf.scala. For most node types, Databricks will set 

spark.executor.memory = (0.8 * container_size)

spark.driver.memory = (0.8 * container_size) - conf.chauffeurMemorySizeMB.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group