topic Re: Data bricks -connect error in Data Engineering

Data bricks -connect error

neeth — Tue, 07 Jan 2025 12:52:02 GMT

Hello,

I new to Databricks and Scala. I created a scala application in my local machine and tried to connect to my cluster in databricks workspace using databricks connect as per the documentation. My cluster is using Databricks runtime version 16.0 (includes Apache Spark 3.5.0, Scala 2.12).

I have added the below dependencies in my build.sbt :

libraryDependencies += "org.apache.spark" %% "spark-sql" % "3.5.1"

libraryDependencies += "org.apache.spark" %% "spark-core" % "3.5.1"

libraryDependencies += "com.databricks" % "databricks-connect" % "16.0.0"

I create spark session by

val spark = DatabricksSession.builder().remote().getOrCreate().

I have a .databrickscfg configuration file with DEFAULT profile with host and cluster_id values.

While running the application , I am getting the below error

Exception in thread "sbt-bg-threads-7" java.lang.NoSuchMethodError: 'org.apache.spark.sql.SparkSession$Builder org.apache.spark.sql.SparkSession$Builder.client(org.apache.spark.sql.connect.client.SparkConnectClient)'

at com.databricks.connect.DatabricksSession$Builder.fromSparkClientConf(DatabricksSession.scala:522)

at com.databricks.connect.DatabricksSession$Builder.fromSdkConfig(DatabricksSession.scala:515)

at com.databricks.connect.DatabricksSession$Builder.getOrCreate(DatabricksSession.scala:446)

at leakagetest.Main$.createSparkSession(Main.scala:41)

at leakagetest.Main$.delayedEndpoint$leakagetest$Main$1(Main.scala:27)

at leakagetest.Main$delayedInit$body.apply(Main.scala:18)

at scala.Function0.apply$mcV$sp(Function0.scala:39)

at scala.Function0.apply$mcV$sp$(Function0.scala:39)

at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)

at scala.App.$anonfun$main$1$adapted(App.scala:80)

at scala.collection.immutable.List.foreach(List.scala:431)

at scala.App.main(App.scala:80)

at scala.App.main$(App.scala:78)

at leakagetest.Main$.main(Main.scala:18)

at leakagetest.Main.main(Main.scala)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:568)

at sbt.Run.invokeMain(Run.scala:144)

at sbt.Run.execute$1(Run.scala:94)

at sbt.Run.$anonfun$runWithLoader$5(Run.scala:121)

at sbt.Run$.executeSuccess(Run.scala:187)

at sbt.Run.runWithLoader(Run.scala:121)

at sbt.Defaults$.$anonfun$bgRunTask$6(Defaults.scala:1988)

at sbt.Defaults$.$anonfun$termWrapper$2(Defaults.scala:1927)

at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)

at scala.util.Try$.apply(Try.scala:213)

at sbt.internal.BackgroundThreadPool$BackgroundRunnable.run(DefaultBackgroundJobService.scala:367)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)

at java.base/java.lang.Thread.run(Thread.java:840)

Could some one please help me to solve this error.

Re: Data bricks -connect error

Walter_C — Tue, 07 Jan 2025 13:36:33 GMT

The error you are encountering, java.lang.NoSuchMethodError: 'org.apache.spark.sql.SparkSession$Builder org.apache.spark.sql.SparkSession$Builder.client(org.apache.spark.sql.connect.client.SparkConnectClient)', suggests that there is a mismatch between the versions of the libraries you are using.

Check Library Versions: Ensure that the versions of spark-core and databricks-connect you are using are compatible with each other. According to the context, the databricks-connect package version must match the Databricks Runtime version. For example, if you are using Databricks Runtime 12.2 LTS, you should use databricks-connect==12.2.*

Re: Data bricks -connect error

neeth — Tue, 07 Jan 2025 13:49:34 GMT

I am using Databricks runtime version 16.0.0, therefore the databricks-connect ==16.0.0 and I changed the spark-core to 3.5.0 as per documentation. Still I am getting the same error.

Re: Data bricks -connect error

Walter_C — Tue, 07 Jan 2025 14:10:20 GMT

If you try with lower DBR does it work or same exact issue?

Re: Data bricks -connect error

neeth — Mon, 13 Jan 2025 14:37:55 GMT

Got the same error with lower DBR.

I am using OAuth user-to-machine (U2M) authentication and a configuration profile named DEFAULT which contains host, clusterId and auth_type saved in .databrickscfg file. I logged in using databricks cli before running the code.

To create SparkSession I used,

 val spark = DatabricksSession.builder().remote().getOrCreate()

On debugging I could see that the, cluster_id ,host ,sdkConfig,token is None.

I followed this tutorial, https://docs.databricks.com/en/dev-tools/databricks-connect/scala/index.html#tutorial

Re: Data bricks -connect error

Walter_C — Mon, 13 Jan 2025 14:51:08 GMT

so the issue is happening on Step 4, is this the only workspace you have synced in the CLI, or you have done works with any other workspace as well

Re: Data bricks -connect error

neeth — Mon, 13 Jan 2025 19:46:54 GMT

This is the only workspace that I have synced.

Re: Data bricks -connect error

Walter_C — Mon, 13 Jan 2025 19:57:57 GMT

Can you try creating another profile instead of the Default one and try with it, it seems that what it is not collecting is the cluster details but wanted to check with a new profile

Re: Data bricks -connect error

neeth — Tue, 14 Jan 2025 07:41:33 GMT

I tried creating another profile and used the below code:

val config = new DatabricksConfig().setProfile("myprofile")

val spark = DatabricksSession.builder().sdkConfig(config).getOrCreate()

I got the same error. While debugging I could see always DatabricksConfig is returned empty.

Re: Data bricks -connect error

saurabh18cs — Tue, 14 Jan 2025 09:59:37 GMT

try this with parameters once:

def get_remote_spark(host: str, cluster_id: str, token: str) -> SparkSession:

from databricks.connect import DatabricksSession

return DatabricksSession.builder.remote(host=host, cluster_id=cluster_id, token=token).getOrCreate()

Run the following command to configure Databricks Connect to use the .databrickscfg file inside CLI:

databricks-connect configure