Data bricks -connect error

neeth
New Contributor III

‎01-07-2025 04:52 AM

Hello, 

I new to Databricks and Scala. I created a scala application in my local machine and tried to connect to my cluster in databricks workspace using databricks connect as per the documentation. My cluster is using Databricks runtime version 16.0 (includes Apache Spark 3.5.0, Scala 2.12). 

I have added the below dependencies in my build.sbt :

libraryDependencies += "org.apache.spark" %% "spark-sql" % "3.5.1"
libraryDependencies += "org.apache.spark" %% "spark-core" % "3.5.1"
libraryDependencies += "com.databricks" % "databricks-connect" % "16.0.0"
 
I create spark session by 
val spark = DatabricksSession.builder().remote().getOrCreate().
I have a .databrickscfg configuration file with DEFAULT profile with host and cluster_id values.
While running the application , I am getting the below error

Exception in thread "sbt-bg-threads-7" java.lang.NoSuchMethodError: 'org.apache.spark.sql.SparkSession$Builder org.apache.spark.sql.SparkSession$Builder.client(org.apache.spark.sql.connect.client.SparkConnectClient)'

at com.databricks.connect.DatabricksSession$Builder.fromSparkClientConf(DatabricksSession.scala:522)

at com.databricks.connect.DatabricksSession$Builder.fromSdkConfig(DatabricksSession.scala:515)

at com.databricks.connect.DatabricksSession$Builder.getOrCreate(DatabricksSession.scala:446)

at leakagetest.Main$.createSparkSession(Main.scala:41)

at leakagetest.Main$.delayedEndpoint$leakagetest$Main$1(Main.scala:27)

at leakagetest.Main$delayedInit$body.apply(Main.scala:18)

at scala.Function0.apply$mcV$sp(Function0.scala:39)

at scala.Function0.apply$mcV$sp$(Function0.scala:39)

at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)

at scala.App.$anonfun$main$1$adapted(App.scala:80)

at scala.collection.immutable.List.foreach(List.scala:431)

at scala.App.main(App.scala:80)

at scala.App.main$(App.scala:78)

at leakagetest.Main$.main(Main.scala:18)

at leakagetest.Main.main(Main.scala)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:568)

at sbt.Run.invokeMain(Run.scala:144)

at sbt.Run.execute$1(Run.scala:94)

at sbt.Run.$anonfun$runWithLoader$5(Run.scala:121)

at sbt.Run$.executeSuccess(Run.scala:187)

at sbt.Run.runWithLoader(Run.scala:121)

at sbt.Defaults$.$anonfun$bgRunTask$6(Defaults.scala:1988)

at sbt.Defaults$.$anonfun$termWrapper$2(Defaults.scala:1927)

at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)

at scala.util.Try$.apply(Try.scala:213)

at sbt.internal.BackgroundThreadPool$BackgroundRunnable.run(DefaultBackgroundJobService.scala:367)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)

at java.base/java.lang.Thread.run(Thread.java:840)

Could some one please help me to solve this error.

0 Kudos