Spark SQL was designed with an optimizer called Catalyst based on the functional programming of Scala. Its two main purposes are: first, to add new optimization techniques to solve some problems with โbig dataโ and second, to allow developers to expand and customize the functions of the optimizer.
