topic Re: Pyspark operations slowness in CLuster 14.3LTS as compared to 13.3 LTS in Data Engineering

Pyspark operations slowness in CLuster 14.3LTS as compared to 13.3 LTS

anish2102 — Sun, 14 Apr 2024 20:37:29 GMT

In my notebook, i am performing few join operations which are taking more than 30s in cluster 14.3 LTS where same operation is taking less than 4s in 13.3 LTS cluster. Can someone help me how can i optimize pyspark operations like joins and withColumn?

Re: Pyspark operations slowness in CLuster 14.3LTS as compared to 13.3 LTS

jose_gonzalez — Thu, 18 Apr 2024 21:48:42 GMT

check the physical query plan for both, DBR 14.3 and 13.3 to compare if these values are different. If they are, then check the Spark UI to identify where did it changed

Re: Pyspark operations slowness in CLuster 14.3LTS as compared to 13.3 LTS

Lakshay — Tue, 23 Apr 2024 14:05:14 GMT

Are you comparing the performance against same dataset?

Re: Pyspark operations slowness in CLuster 14.3LTS as compared to 13.3 LTS

anish2102 — Tue, 23 Apr 2024 15:10:43 GMT

I have found the issue. It was actually with code where dataframe was being referred multiple times in withcolumn and join operations in form dataframe['col_name'] which is creating more than 20 spark jobs and hence causing degradation in performance of notebook. If i refer column using col() function in both join and withcolumn hen it is working fast compared to previous one. Also it is crating 1 or 2 spark job only.

Re: Pyspark operations slowness in CLuster 14.3LTS as compared to 13.3 LTS

Lakshay — Tue, 23 Apr 2024 15:42:23 GMT

Thank you for sharing the analysis