I got this question from some customers and I want ti clarify here too
I think we are conflating two things:
- Catalyst optimizer is about coming up "Steps to take to execute the query". For example, the optimizer will decide how and when to do the join, aggregations, filters etc. When it should do what. This is also called "Physical Plan" in technical parlance. (Catalyst optimizer is more for logical planning but that is a finer detail)
- The execution engine is actually the engine that carries out the steps decided by the optimizer. It does not second guess the steps, but rather makes sure to carry them out as efficiently as possible.
Delta Engine (or Photon) is an execution engine and not an optimizer. It is more directly comparable with the Tungsten (whole stage code generation) execution engine currently available in open source Spark. Both Tungsten and Delta-Engine are execution engines.
