In my notebook, i am performing few join operations which are taking more than 30s in cluster 14.3 LTS where same operation is taking less than 4s in 13.3 LTS cluster. Can someone help me how can i optimize pyspark operations like joins and withColumn?
I have found the issue. It was actually with code where dataframe was being referred multiple times in withcolumn and join operations in form dataframe['col_name'] which is creating more than 20 spark jobs and hence causing degradation in performance of notebook. If i refer column using col() function in both join and withcolumn hen it is working fast compared to previous one. Also it is crating 1 or 2 spark job only.
check the physical query plan for both, DBR 14.3 and 13.3 to compare if these values are different. If they are, then check the Spark UI to identify where did it changed
I have found the issue. It was actually with code where dataframe was being referred multiple times in withcolumn and join operations in form dataframe['col_name'] which is creating more than 20 spark jobs and hence causing degradation in performance of notebook. If i refer column using col() function in both join and withcolumn hen it is working fast compared to previous one. Also it is crating 1 or 2 spark job only.
Welcome to Databricks Community: Lets learn, network and celebrate together
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
