Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-05-2024 10:42 PM
Hello! It's possible that the increase in runtime when upgrading from Spark 3.3.0 (DBR 11.3) to Spark 3.4.1 (DBR 13.3) is due to changes in the underlying R runtime or package versions. When you upgrade to a new version of Spark, the R packages that you use may also be updated, which can affect their performance.
To determine whether the issue is related to the R packages, you can try creating a new zip file of the R packages using DBR 13.3 and then using that zip file in your job. This will ensure that the R packages are compatible with the new version of Spark.
If the job still takes a long time to complete, there may be other factors contributing to the increase in runtime. In this case, you can try profiling your code to identify any performance bottlenecks.
Additionally, you can try adjusting the Spark configuration settings to optimize the performance of your job. For example, you can try increasing the number of partitions or adjusting the memory settings.
I hope this helps!
