Sparklyr error in spark_apply: Error: java.lang.NoSuchMethodError

[omjohn]

When trying incorporate an R package into my Spark workflow using the spark_apply() funciton in Sparklyr, I get the error:

Error: java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.encoders.RowEncoder$.apply(Lorg/apache/spark/sql/types/StructType;)Lorg/apache/spark/sql/catalyst/encoders/ExpressionEncoder;

I simplied to an easy test case and still happens with a simple chunk that should run fine:

data %>%
  sparklyr::spark_apply(
    function(data) {data * 10})

My connection is:
sc <- sparklyr::spark_connect(method = "databricks")

Internet seems to suggest this happens due to conflicts between a package and the curerrent version of Spark, though I haven't seen discussions particuarly applied to sparklyr. I'm running version 1.8.1 of sparklyr, tried loading  from both CRAN and github. Using Databricks Runtime 14.3.x-photon-scala2.12. Other info:

R version 4.3.1 (2023-06-16)
Platform: x86_64-pc-linux-gnu (64-bit) Running under:
Ubuntu 22.04.4 LTS

I'm a data scientist, not an engineer, so any advice would be helpful on how to get this running. Thanks.

[Kaniz_Fatma]

Hi @omjohn, To resolve this, first check the compatibility matrix on the [sparklyr GitHub page](https://github.com/sparklyr/sparklyr#compatibility) to ensure your `sparklyr` version matches the Spark version. Update `sparklyr` to the latest version from CRAN or GitHub, specify the Spark version explicitly when connecting, and ensure there are no conflicting packages by running in a clean R session. If needed, try a different Databricks runtime or set custom Spark configurations.