Runtime increases exponentially from 11.3 to 13.3

[simple89]

Hello. I am using R on databricks and using the below approach. 

My Spark version:

Single node: i3.2xlarge · On-demand · DBR: 11.3 LTS (includes Apache Spark 3.3.0, Scala 2.12) · us-east-1a, the job takes 1 hour

I install all R packages (including a geospatial package terra) in my notebook and zip the installed R packages so that I don't have to install the packages again and again. 

I deploy a job which does following:

1. Get the zip R packages and unzip

2. load the library 

3. do stuff

The job takes an hour to complete. 

However, when I update the Spark  to below, my run times increase exponentially. 
Single node: i3.2xlarge · On-demand · DBR: 13.3 LTS (includes Apache Spark 3.4.1, Scala 2.12) · us-east-1a

I am not a spark expert but why is changing 11.3 to 13.3 increases the run time? Would the ideal solution be that I create the zip packages again but using the 13.3 instead of 11.3?

[Sidhant07]

Hello! It's possible that the increase in runtime when upgrading from Spark 3.3.0 (DBR 11.3) to Spark 3.4.1 (DBR 13.3) is due to changes in the underlying R runtime or package versions. When you upgrade to a new version of Spark, the R packages that you use may also be updated, which can affect their performance.

To determine whether the issue is related to the R packages, you can try creating a new zip file of the R packages using DBR 13.3 and then using that zip file in your job. This will ensure that the R packages are compatible with the new version of Spark.

If the job still takes a long time to complete, there may be other factors contributing to the increase in runtime. In this case, you can try profiling your code to identify any performance bottlenecks.

Additionally, you can try adjusting the Spark configuration settings to optimize the performance of your job. For example, you can try increasing the number of partitions or adjusting the memory settings.

I hope this helps!