@VZLA/@Walter_C   Thanks for the quick answers! I understand that the Photon engine requires memory for its optimization tasks, and this memory usage impacts the executor memory.

I’ve got few more questions, and I’d really appreciate it if you could help me out.

1. Is the memory allocated to the Photon engine fixed, or is it based on a percentage of the node’s total memory?
2. How can I calculate the value of spark.executor.memory based on a specific node type? I’ve gone through some articles to understand Spark's memory allocation, but the results don’t match the spark.executor.memory value set by Databricks.
3. I need clarification on how memory is allocated and the memory values displayed on different tabs of the Databricks Spark UI. Below are the configuration values for my node type, Standard_DS4_v2 (28GB RAM, 8 cores).
   
   1. What does the 'Storage Memory' column in the Spark UI -> Executors represent? In my case, it shows 9.4GB. I assume this is half of 18409m, so does it indicate only the storage memory portion of the executor, which is 50% of the total executor memory? If so, can I conclude that the remaining 9.4GB is used for execution memory?
   2. What is spark.executor.memory (18409m = 17.97GB)? How can I calculate this value based on a specific node type X (similar to question 2 asked above)?
   3. What does the 'Memory' column in the Spark compute UI - Master -> Workers represent? It's showing 22.5GiB (18.0GiB used). I assume 18.0GiB corresponds to 18409m, but what does the 22.5GiB indicate, considering the node memory is 28GB?
   4. What does the 'Memory per Executor' column in the Spark compute UI - Master -> Running Applications refer to? It shows 18409m. Is this the same as the value in question 2.