Hi @dbuserng ,
The free -h
command in the web terminal shows only 8.9GB of available memory on your driver, which is a Standard_DS3_v2 instance with 14GB of memory, because Databricks has services running on each node. This means the maximum allowable memory for Spark is less than the total memory capacity of the VM reported by the cloud provider. Even if spark.driver.memory
is not explicitly set, some memory is reserved for these services, reducing the amount available for Spark processes.

As a preface, the maximum memory size given to Spark is constrained by the other internal services we run on the VM. The current formula for this is
container_size = (vm_size * 0.97 - 4800MB)
and is implicitly defined in NodeTypeConf.scala
. For most node types, Databricks will set
spark.executor.memory = (0.8 * container_size)
spark.driver.memory = (0.8 * c
ontainer_size) - conf.chauffeurMemorySizeMB
.