Error AgnosticEncoder.isStruct() in Intellij using Scala locally.

[FhSpZ]

I've been trying to execute a connect to Azure Databricks from Intellij using Scala locally, but I've got this error below:

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.encoders.AgnosticEncoder.isStruct()Z

at org.apache.spark.sql.connect.client.arrow.ArrowDeserializers$.deserializerFor(ArrowDeserializer.scala:78)

at org.apache.spark.sql.connect.client.arrow.ArrowDeserializingIterator.<init>(ArrowDeserializer.scala:534)

at org.apache.spark.sql.connect.client.SparkResult$$anon$1.initialize(SparkResult.scala:252)

at org.apache.spark.sql.connect.client.SparkResult$$anon$1.hasNext(SparkResult.scala:257)

at org.apache.spark.sql.connect.client.SparkResult.toArray(SparkResult.scala:206)

at org.apache.spark.sql.Dataset.$anonfun$collect$1(Dataset.scala:2830)

at org.apache.spark.sql.Dataset.withResult(Dataset.scala:3275)

at org.apache.spark.sql.Dataset.collect(Dataset.scala:2829)

at org.apache.spark.sql.Dataset.count(Dataset.scala:2866)

at MainSpark$.delayedEndpoint$MainSpark$1(MainSpark.scala:32)

at MainSpark$delayedInit$body.apply(MainSpark.scala:4)

at scala.Function0.apply$mcV$sp(Function0.scala:39)

at scala.Function0.apply$mcV$sp$(Function0.scala:39)

at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)

at scala.App.$anonfun$main$1$adapted(App.scala:80)

at scala.collection.immutable.List.foreach(List.scala:431)

at scala.App.main(App.scala:80)

at scala.App.main$(App.scala:78)

at MainSpark$.main(MainSpark.scala:4)

at MainSpark.main(MainSpark.scala)

This error happened when is tried to run a show() command for example, but a simple printSchema() works. I have no idea why this happen.

Cluster DB Info:

13.3 LTS (includes Apache Spark 3.4.1, Scala 2.12) - Standard_D4s_v3

sbt file:

ThisBuild / version := "0.1.0-SNAPSHOT"

ThisBuild / scalaVersion := "2.12.15"

lazy val root = (project in file("."))
  .settings(
    name := "ProjSparkScala"
  )
libraryDependencies ++= Seq(
  "org.apache.commons" % "commons-lang3" % "3.14.0",
  "org.apache.spark" %% "spark-core" % "3.4.1",
  "org.apache.spark" %% "spark-sql" % "3.4.1",
  "org.apache.spark" %% "spark-streaming" % "3.4.1",
  "com.databricks" % "databricks-connect" % "13.3.1",
  "com.databricks" % "databricks-sdk-java" % "0.0.1",
  "com.databricks" %% "databricks-dbutils-scala" % "0.1.4",
  "io.delta" %% "delta-spark" % "3.1.0",
  "io.delta" %% "delta-core" % "2.4.0",
  "org.apache.spark" %% "spark-catalyst" % "3.4.1"
)

Any idea?

[FhSpZ]

Hi @Retired_mod,

I ensured that I was using the correct Spark version that matched the version of my databricks runtime, which was the same. But I tried use the Spark version 3.5.1 locally in the .sbt dependencies, then this worked, kind strange.

Anyway thank you very much @Retired_mod, your comment was very helpfully.