โ02-03-2022 01:55 AM
I want to use databricks inside vscode and I therefore need Databricks-connect
I configure my settings using
databricks-connect configure
as follows:
Databricks Host [https://adb-1409757184094616.16.azuredatabricks.net]
Databricks Token [<my token>]
Cluster ID [1220-124223-ku6xm034]
Org ID [<my organization>]
Port [5001]
following @Kaniz Fatmaโ and @Arjun Kaimaparambil Rajanโ comments here is some other information that might be relevant:
These are my Spark config settings of the cluster I want to run in vscode
spark.databricks.cluster.profile serverless
spark.databricks.service.port 8787
spark.sql.legacy.allowNonEmptyLocationInCTAS = true
spark.databricks.service.server.enabled true
spark.databricks.repl.allowedLanguages sql,python,r
when I run
databricks-connect test
in my prompt I get the following error:
ERROR SparkClientManager: Fail to get the SparkClient
java.util.concurrent.ExecutionException: com.databricks.service.SparkServiceConnectionException: Invalid shard address: "https://adb-1409757184094616.16.azuredatabricks.net/"
The stacktrace is attached to this question.
I tried to change my shard address to
I discussed this with my data engineer at the company and he told me that it could be an issue with the private endpoints that they installed last year. This might be blocking the connection with my remote pc. Anyone any ideas?
โ03-11-2022 03:06 AM
โ02-03-2022 04:33 AM
Hi @ignace decocqโ , You need to check the version of all the tools.
Are you behind a proxy or a network that may have a layer 7 firewall?
Have you set:
spark.databricks.service.server.enabled true
spark.databricks.service.port 8787
โ02-03-2022 04:40 AM
what I have configured is the following:
cluster is 9.1 LTS ML and my python version is 3.8.10 in the base conda environment
I did had the server enabled but did not set the port to 8787 in the cluster settings.
I tried setting the port of the cluster to 8787 while keeping the server enabled but it was no good. Still got the same error.
I do not know if I have a 7 layer firewall. Is there a way to see it?
โ02-09-2022 06:04 AM
Hi @Arjun Kaimaparambil Rajanโ , you can find the stacktrace in the attechment of the question. By REST API you mean if I can connect to the databricks workspace via the webbrowser ? (which I can)
Hopefully you can help me with this extra information. already many thanks for the support.
โ02-14-2022 05:59 AM
@Arjun Kaimaparambil Rajanโ I followed the link you send me and when I run :
I get the following exception:
javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
at sun.security.ssl.SSLSocketImpl.handleEOF(Unknown Source)
at sun.security.ssl.SSLSocketImpl.decode(Unknown Source)
at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source)
at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
at sun.security.ssl.SSLSocketImpl.ensureNegotiated(Unknown Source)
at sun.security.ssl.SSLSocketImpl.access$200(Unknown Source)
at sun.security.ssl.SSLSocketImpl$AppOutputStream.write(Unknown Source)
at sun.security.ssl.SSLSocketImpl$AppOutputStream.write(Unknown Source)
at SSLPoke.main(SSLPoke.java:23)
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at sun.security.ssl.SSLSocketInputRecord.read(Unknown Source)
at sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source)
at sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source)
at sun.security.ssl.SSLTransport.decode(Unknown Source)
... 8 more
I tried solving this exception following this video but to no avail. I am really feeling that I am shooting in the dark for the moment. Hopefully you can help me with that.
โ03-11-2022 03:06 AM
โ03-14-2022 05:10 PM
Hi @ignace decocqโ
Did @Kaniz Fatmaโ response help you to resolved your question? if yes, please mark it as best response. If not, please let us know and we will continue helping you to find a solution.
โ02-09-2022 07:44 AM
@ignace decocqโ - I noticed that you edited your original post. I'm not a technical person, so I can't tell. Did your change make the original answers not make sense anymore? If they did, I can add the information back in there. I promise I won't disrupt the current edit. Just let me know. ๐
โ02-09-2022 08:32 AM
Hey @Piper Wilsonโ yes, I indeed edited my post. I did that to make the question more wholesome: If your question is answered with a question, then your initial question was not clear enough. That is why I edited the post.
I do not know what you want to add but if it makes the conversation more clear, go ahead.
โ03-29-2022 07:54 AM
Hey everyone,
I reinstalled everything just like on the website in data thirst.
When I reinstalled it, I got a message that the firewall was blocking certain features of java 8.
As this is a company's pc I could not alter the settings of the firewall. however I went to the IT department and they disabled the company policy protection of my pc. This is basically a Software called Cloudfare. Once this was disabled everything worked fine.
I probably should solve this now within the company.
Thanks for the help!
โ06-17-2022 10:06 AM
In case it helps anyone, I ran into this issue and had to remove the trailing / from the host name. It used to work fine with the trailing / so something must have changed.
โ06-23-2022 06:57 AM
It helped, THANK YOU @Justin Allredโ !. Our problems started about the same time as you posted the above. Would not have easily guessed the solution otherwise.
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group