@Einsatz thanks for your question!

1) Why does enabling Photon reduce the executor memory?
Photon allocates a significant portion of memory off-heap for its C++ engine. As a result, the on-heap memory (shown by spark.executor.memory) appears lower once Photon is enabled.

2) Is there a way to keep spark.executor.memory at 18409m when Photon is enabled?
Not directly. You must either increase your node’s total memory (e.g., choose a larger instance type) or adjust off-heap allocations to accommodate Photon’s requirements.

Photon is a separate C++ engine embedded within Spark to accelerate certain SQL workloads. So, it requires its own memory space, therefore you can either provision extra memory for Photon or run those queries in the regular Spark engine with full on-heap capacity. It comes with a cost which you need to balance and account accordingly.