Driver: how much memory is actually available?

dbuserng — Wed, 22 Jan 2025 14:48:24 GMT

I have a cluster where Driver type is Standard_DS3_v2 (14GB Memory and 4 Cores). When I use free -h command in Web terminal (see attached screenshot) I get the response that I only have 8.9GB memory available on my driver - why is that?

fyi, spark.driver.memory is not set

Re: Driver: how much memory is actually available?

Sidhant07 — Thu, 30 Jan 2025 08:17:58 GMT

Hi @dbuserng ,

The free -h command in the web terminal shows only 8.9GB of available memory on your driver, which is a Standard_DS3_v2 instance with 14GB of memory, because Databricks has services running on each node. This means the maximum allowable memory for Spark is less than the total memory capacity of the VM reported by the cloud provider. Even if spark.driver.memory is not explicitly set, some memory is reserved for these services, reducing the amount available for Spark processes.

As a preface, the maximum memory size given to Spark is constrained by the other internal services we run on the VM. The current formula for this is

container_size = (vm_size * 0.97 - 4800MB)

and is implicitly defined in NodeTypeConf.scala. For most node types, Databricks will set

spark.executor.memory = (0.8 * container_size)

spark.driver.memory = (0.8 * container_size) - conf.chauffeurMemorySizeMB.

topic Re: Driver: how much memory is actually available? in Administration & Architecture

Driver: how much memory is actually available?

Re: Driver: how much memory is actually available?