Databricks allocates resources to executors on a node based on several factors, and it appears that your cluster configuration is using default settings since no specific Spark configurations were provided.

1. Executor Memory Allocation:

   • The spark.executor.memory setting you observed (24G) is the amount of memory allocated to each executor. This is typically a fraction of the total node memory to ensure there is enough overhead for the operating system and other processes.
   • The observed memory usage cap of 48G per worker node suggests that there might be two executors per node, each using 24G of memory.

2. Executor Cores:

   • The absence of spark.executor.cores in the Spark UI environment tab indicates that the default configuration is being used. By default, Databricks assigns one executor per worker node, and the number of cores per executor is determined by the total number of cores available on the node divided by the number of executors.
   • For a node with 16 cores, if there are two executors, each executor would typically use 8 cores.

3. Resource Allocation:

   • Databricks runs one executor per worker node by default, but this can be adjusted by specifying the spark.executor.instances and spark.executor.cores configurations.
   • The total memory available to executors on a node is less than the node's total memory to leave room for system processes and Spark's overhead.