The memory allocated to the Photon engine is not fixed; it is based on a percentage of the node’s total memory.

To calculate the value of spark.executor.memory based on a specific node type, you can use the following formula:

container_size = (vm_size * 0.97 - 4800MB)
spark.executor.memory = (0.8 * container_size)
For your node type, Standard_DS4_v2 (28GB RAM, 8 cores), the calculation would be: container_size = (28GB * 0.97 - 4800MB)
spark.executor.memory = (0.8 * container_size)
This results in approximately 17.97GB (18409m).

Regarding the 'Storage Memory' column in the Spark UI -> Executors, it represents the amount of memory allocated for storage (caching) within the executor. In your case, it shows 9.4GB, which is half of the total executor memory (18409m). This indicates that 50% of the total executor memory is allocated for storage memory, and the remaining 50% is used for execution memory.

The 'Memory' column in the Spark compute UI - Master -> Workers represents the total memory allocated to the worker node. The 22.5GiB (18.0GiB used) indicates that 18.0GiB corresponds to the spark.executor.memory value (18409m), and the remaining memory is used by other processes and overheads.

The 'Memory per Executor' column in the Spark compute UI - Master -> Running Applications refers to the memory allocated per executor, which in your case is 18409m. This is the same value as the spark.executor.memory calculated above