topic Re: databricks-connect version 13: spark-class2.cmd not found in Data Engineering

databricks-connect version 13: spark-class2.cmd not found

Lazloo — Mon, 08 May 2023 09:00:59 GMT

I install the newest version "databricks-connect==13.0.0". Now get the issue

Command C:\Users\Y\AppData\Local\pypoetry\Cache\virtualenvs\X-py3.9\Lib\site-packages\pyspark\bin\spark-class2.cmd"" not found

konnte nicht gefunden werden.

Traceback (most recent call last):

File "C:\X\repositories\schema-integration-customer\tmp_run_builder.py", line 37, in <module>

spark = get_spark()

File "C:\X\repositories\data-common\X\da\common\_library\spark.py", line 60, in get_spark

return builder.getOrCreate()

File "C:\Users\Y\AppData\Local\pypoetry\Cache\virtualenvs\X-schema-integration-customer-hjO9aLIy-py3.9\lib\site-packages\pyspark\sql\session.py", line 479, in getOrCreate

else SparkContext.getOrCreate(sparkConf)

File "C:\Users\Y\AppData\Local\pypoetry\Cache\virtualenvs\X-schema-integration-customer-hjO9aLIy-py3.9\lib\site-packages\pyspark\context.py", line 560, in getOrCreate

SparkContext(conf=conf or SparkConf())

File "C:\Users\y\AppData\Local\pypoetry\Cache\virtualenvs\x-schema-integration-customer-hjO9aLIy-py3.9\lib\site-packages\pyspark\context.py", line 202, in __init__

SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)

File "C:\Users\y\AppData\Local\pypoetry\Cache\virtualenvs\x-schema-integration-customer-hjO9aLIy-py3.9\lib\site-packages\pyspark\context.py", line 480, in _ensure_initialized

SparkContext._gateway = gateway or launch_gateway(conf)

File "C:\Users\y\AppData\Local\pypoetry\Cache\virtualenvs\x-schema-integration-customer-hjO9aLIy-py3.9\lib\site-packages\pyspark\java_gateway.py", line 106, in launch_gateway

raise RuntimeError("Java gateway process exited before sending its port number")

RuntimeError: Java gateway process exited before sending its port number

Process finished with exit code 1

I use Windows and for the version "databricks-connect==11.3.10" everything run smooth

Re: databricks-connect version 13: spark-class2.cmd not found

Anonymous — Sat, 13 May 2023 15:38:09 GMT

@Lazloo XP :

The error message you received indicates that there is a problem with launching the Java gateway process. This is typically caused by a misconfiguration in the environment variables that point to the location of the Spark and Java executables.

To resolve this issue, you can try the following steps:

Verify that you have installed the correct version of Java and Spark that is compatible with Databricks Connect 13.0.0.
Check that the environment variables JAVA_HOME and SPARK_HOME are set correctly and point to the correct directories where Java and Spark are installed.
Make sure that the bin directories of both Java and Spark are included in the PATH environment variable.
Ensure that there are no conflicting versions of Java or Spark installed on your system that may be causing conflicts.

If you are still encountering issues after verifying these steps, you may want to try reinstalling Databricks Connect 13.0.0 or rolling back to version 11.3.10 if it was working previously.

Re: databricks-connect version 13: spark-class2.cmd not found

Lazloo — Wed, 17 May 2023 06:41:57 GMT

Hey @Suteja Kanuri ,

I have no issue with version 11.3.10.. Therefore I believe that my environment variables are set correctly.

Re: databricks-connect version 13: spark-class2.cmd not found

Lazloo — Wed, 17 May 2023 11:30:01 GMT

with the newest Version the error changed to

```

Traceback (most recent call last):

File "C:\x\repositories\lf_backup_repo\snippets.py", line 4, in <module>

spark = SparkSession.builder.getOrCreate()

File "C:\Users\x\AppData\Local\pypoetry\Cache\virtualenvs\lf-privat-eCptrNhE-py3.8\lib\site-packages\pyspark\sql\session.py", line 469, in getOrCreate

raise RuntimeError(

RuntimeError: Only remote Spark sessions using Databricks Connect are supported. Could not find connection parameters to start a Spark remote session.

```

Re: databricks-connect version 13: spark-class2.cmd not found

Hardy — Wed, 31 May 2023 05:21:28 GMT

I am also facing similar issue with databricks-connect 13.

Getting RuntimeError: Only remote Spark sessions using Databricks Connect are supported. Could not find connection parameters to start a Spark remote session.

Databricks-connect - 13

DBR - 13

Python: 3.10.11

Re: databricks-connect version 13: spark-class2.cmd not found

HD — Sat, 28 Oct 2023 19:09:07 GMT

I get the same error. Please help with any hints.

Re: databricks-connect version 13: spark-class2.cmd not found

Susumu_Asaga — Tue, 30 Apr 2024 14:21:09 GMT

Use this code:

from databricks.connect import DatabricksSession spark = DatabricksSession.builder.getOrCreate()