https://community.databricks.com/t5/data-engineering/unexpected-performance-behaviors-due-to-changes-in-the-spark/m-p/45174#M27803 <P>Hi!<BR /><BR />We have recently upgraded our cluster from&nbsp;<SPAN>Databricks Runtime 10.4 LTS which includes Apache Spark 3.2.1 to to Databricks Runtime 13.3 LTSincludes Apache Spark 3.2.1&nbsp;powered by Apache Spark 3.3.0 and noticed that one of our jobs runtime has dramatically increased (actually it was terminated before it even finished).<BR /></SPAN></P><P><SPAN>On 3.2.1 the job would run aprox. 40 mins, while on 3.3.0 it was terminated after 3 hours. According to driver logs, it stuck on one stage which had 160000 tasks (TaskSetManager: Finished task 18443.0 in stage 204.0 in 8746 ms on (executor 1) (17678/160000)) while the run on 3.2.1 would never reach that number - max. a few hundred tasks. Also worth mentioning that disk was expanding to the hights I never seen.<BR /><BR />I was able to find the function where behaviour of 3.3.0 has changed compared to 3.2.1. It is <STRONG>crossjoin</STRONG>.</SPAN></P><P>Has anyone experienced such situation or maybe have suggestion why it could have happened? I have a feeling that it has something to do with new or changed default spark properties, but could not identify which ones.<BR /><BR />Also,&nbsp;spark.conf.set("spark.databricks.io.cache.enabled", "true").</P> Sun, 17 Sep 2023 20:37:28 GMT Direo 2023-09-17T20:37:28Z https://community.databricks.com/t5/data-engineering/unexpected-performance-behaviors-due-to-changes-in-the-spark/m-p/45174#M27803 <P>Hi!<BR /><BR />We have recently upgraded our cluster from&nbsp;<SPAN>Databricks Runtime 10.4 LTS which includes Apache Spark 3.2.1 to to Databricks Runtime 13.3 LTSincludes Apache Spark 3.2.1&nbsp;powered by Apache Spark 3.3.0 and noticed that one of our jobs runtime has dramatically increased (actually it was terminated before it even finished).<BR /></SPAN></P><P><SPAN>On 3.2.1 the job would run aprox. 40 mins, while on 3.3.0 it was terminated after 3 hours. According to driver logs, it stuck on one stage which had 160000 tasks (TaskSetManager: Finished task 18443.0 in stage 204.0 in 8746 ms on (executor 1) (17678/160000)) while the run on 3.2.1 would never reach that number - max. a few hundred tasks. Also worth mentioning that disk was expanding to the hights I never seen.<BR /><BR />I was able to find the function where behaviour of 3.3.0 has changed compared to 3.2.1. It is <STRONG>crossjoin</STRONG>.</SPAN></P><P>Has anyone experienced such situation or maybe have suggestion why it could have happened? I have a feeling that it has something to do with new or changed default spark properties, but could not identify which ones.<BR /><BR />Also,&nbsp;spark.conf.set("spark.databricks.io.cache.enabled", "true").</P> Sun, 17 Sep 2023 20:37:28 GMT https://community.databricks.com/t5/data-engineering/unexpected-performance-behaviors-due-to-changes-in-the-spark/m-p/45174#M27803 Direo 2023-09-17T20:37:28Z https://community.databricks.com/t5/data-engineering/unexpected-performance-behaviors-due-to-changes-in-the-spark/m-p/45218#M27817 <P>Seems that broadcasting the smaller table in crossjoin did the magic.</P> Mon, 18 Sep 2023 10:24:37 GMT https://community.databricks.com/t5/data-engineering/unexpected-performance-behaviors-due-to-changes-in-the-spark/m-p/45218#M27817 Direo 2023-09-18T10:24:37Z