Sparklyr error in spark_apply: Error: java.lang.NoSuchMethodError

omjohn — Mon, 05 Aug 2024 01:56:25 GMT

When trying incorporate an R package into my Spark workflow using the spark_apply() funciton in Sparklyr, I get the error:

Error: java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.encoders.RowEncoder$.apply(Lorg/apache/spark/sql/types/StructType;)Lorg/apache/spark/sql/catalyst/encoders/ExpressionEncoder;

I simplied to an easy test case and still happens with a simple chunk that should run fine:

data %>%
sparklyr::spark_apply(
function(data) {data * 10})

My connection is:
sc <- sparklyr::spark_connect(method = "databricks")

Internet seems to suggest this happens due to conflicts between a package and the curerrent version of Spark, though I haven't seen discussions particuarly applied to sparklyr. I'm running version 1.8.1 of sparklyr, tried loading from both CRAN and github. Using Databricks Runtime 14.3.x-photon-scala2.12. Other info:

R version 4.3.1 (2023-06-16)
Platform: x86_64-pc-linux-gnu (64-bit) Running under:
Ubuntu 22.04.4 LTS

I'm a data scientist, not an engineer, so any advice would be helpful on how to get this running. Thanks.

Re: Sparklyr error in spark_apply: Error: java.lang.NoSuchMethodError

Vidhi_Khaitan — Mon, 14 Jul 2025 08:35:07 GMT

Hi @omjohn
Can you try downgrading to a Databricks Runtime 13.3 LTS, which uses Spark 3.4.x, officially supported by sparklyr 1.8.1. I believe it would provide a more stable and better-tested integration.

topic Re: Sparklyr error in spark_apply: Error: java.lang.NoSuchMethodError in Warehousing & Analytics

Sparklyr error in spark_apply: Error: java.lang.NoSuchMethodError

Re: Sparklyr error in spark_apply: Error: java.lang.NoSuchMethodError