https://community.databricks.com/t5/data-engineering/what-are-the-four-phase-of-transformation-where-catalyst/m-p/19602#M13158 <P>1. Analysis</P><P>The first phase of Spark SQL optimization is the analysis. Spark SQL starts with a relationship to be processed that can be in two ways. A serious form from an AST (abstract syntax tree) returned by an SQL parser, and on the other hand from a DataFrame object of the Spark SQL API.</P><P>2.&nbsp;Logic Optimization Plan</P><P>The second phase is the logical optimization plan. In this phase, rule-based optimization is applied to the logical plan. It is possible to easily add new rules.</P><P>3. Physical plan</P><P>In the physical plan phase, Spark SQL takes the logical plan and generates one or more physical plans using the physical operators that match the Spark execution engine. The plan to be executed is selected using the cost-based model (comparison between model costs).</P><P>4. Code generation</P><P>Code generation is the final phase of optimizing Spark SQL. To run on each machine, it is necessary to generate Java code bytecode.</P> Fri, 25 Jun 2021 16:10:26 GMT User16826994223 2021-06-25T16:10:26Z https://community.databricks.com/t5/data-engineering/what-are-the-four-phase-of-transformation-where-catalyst/m-p/19601#M13157 Fri, 25 Jun 2021 16:10:05 GMT https://community.databricks.com/t5/data-engineering/what-are-the-four-phase-of-transformation-where-catalyst/m-p/19601#M13157 User16826994223 2021-06-25T16:10:05Z https://community.databricks.com/t5/data-engineering/what-are-the-four-phase-of-transformation-where-catalyst/m-p/19602#M13158 <P>1. Analysis</P><P>The first phase of Spark SQL optimization is the analysis. Spark SQL starts with a relationship to be processed that can be in two ways. A serious form from an AST (abstract syntax tree) returned by an SQL parser, and on the other hand from a DataFrame object of the Spark SQL API.</P><P>2.&nbsp;Logic Optimization Plan</P><P>The second phase is the logical optimization plan. In this phase, rule-based optimization is applied to the logical plan. It is possible to easily add new rules.</P><P>3. Physical plan</P><P>In the physical plan phase, Spark SQL takes the logical plan and generates one or more physical plans using the physical operators that match the Spark execution engine. The plan to be executed is selected using the cost-based model (comparison between model costs).</P><P>4. Code generation</P><P>Code generation is the final phase of optimizing Spark SQL. To run on each machine, it is necessary to generate Java code bytecode.</P> Fri, 25 Jun 2021 16:10:26 GMT https://community.databricks.com/t5/data-engineering/what-are-the-four-phase-of-transformation-where-catalyst/m-p/19602#M13158 User16826994223 2021-06-25T16:10:26Z