Databricks-connect invalid shard address

[korilium]

I want to use databricks inside vscode and I therefore need Databricks-connect

I configure my settings using

databricks-connect configure

as follows:

Databricks Host [https://adb-1409757184094616.16.azuredatabricks.net]

Databricks Token [<my token>]

Cluster ID [1220-124223-ku6xm034]

Org ID [<my organization>]

Port [5001]

following @Kaniz Fatma​ and @Arjun Kaimaparambil Rajan​ comments here is some other information that might be relevant:

• databricks runtime version (cluster): 9.1 LTS ML
• python version in conda (base) environment: 3.8.10
• Check certificate issues

These are my Spark config settings of the cluster I want to run in vscode

spark.databricks.cluster.profile serverless
spark.databricks.service.port 8787
spark.sql.legacy.allowNonEmptyLocationInCTAS = true
spark.databricks.service.server.enabled true
spark.databricks.repl.allowedLanguages sql,python,r

when I run

databricks-connect test

in my prompt I get the following error:

ERROR SparkClientManager: Fail to get the SparkClient
 
java.util.concurrent.ExecutionException: com.databricks.service.SparkServiceConnectionException: Invalid shard address: "https://adb-1409757184094616.16.azuredatabricks.net/"

The stacktrace is attached to this question.

I tried to change my shard address to

• Databricks Host [https://northeurope.azuredatabricks.net]
• Databricks Host [https://westeurope.azuredatabricks.net]

I discussed this with my data engineer at the company and he told me that it could be an issue with the private endpoints that they installed last year. This might be blocking the connection with my remote pc. Anyone any ideas?

[Kaniz]

Hi @ignace decocq​ , You need to check the version of all the tools.

Are you behind a proxy or a network that may have a layer 7 firewall?

Have you set:

spark.databricks.service.server.enabled true
 
spark.databricks.service.port 8787
 

[korilium]

what I have configured is the following:

cluster is 9.1 LTS ML and my python version is 3.8.10 in the base conda environment

I did had the server enabled but did not set the port to 8787 in the cluster settings.

I tried setting the port of the cluster to 8787 while keeping the server enabled but it was no good. Still got the same error.

I do not know if I have a 7 layer firewall. Is there a way to see it?

[korilium]

Hi @Arjun Kaimaparambil Rajan​ , you can find the stacktrace in the attechment of the question. By REST API you mean if I can connect to the databricks workspace via the webbrowser ? (which I can)

Hopefully you can help me with this extra information. already many thanks for the support.

[korilium]

@Arjun Kaimaparambil Rajan​ I followed the link you send me and when I run :

I get the following exception:

javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
        at sun.security.ssl.SSLSocketImpl.handleEOF(Unknown Source)
        at sun.security.ssl.SSLSocketImpl.decode(Unknown Source)
        at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source)
        at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
        at sun.security.ssl.SSLSocketImpl.ensureNegotiated(Unknown Source)
        at sun.security.ssl.SSLSocketImpl.access$200(Unknown Source)
        at sun.security.ssl.SSLSocketImpl$AppOutputStream.write(Unknown Source)
        at sun.security.ssl.SSLSocketImpl$AppOutputStream.write(Unknown Source)
        at SSLPoke.main(SSLPoke.java:23)
Caused by: java.io.EOFException: SSL peer shut down incorrectly
        at sun.security.ssl.SSLSocketInputRecord.read(Unknown Source)
        at sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source)
        at sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source)
        at sun.security.ssl.SSLTransport.decode(Unknown Source)
        ... 8 more

I tried solving this exception following this video but to no avail. I am really feeling that I am shooting in the dark for the moment. Hopefully you can help me with that.

[Kaniz · Accepted solution]

Hi @ignace decocq​ , Please go through the steps provided in the link. Thank you.

[jose_gonzalez]

Hi @ignace decocq​ 

Did @Kaniz Fatma​ response help you to resolved your question? if yes, please mark it as best response. If not, please let us know and we will continue helping you to find a solution.

[Anonymous]

@ignace decocq​ - I noticed that you edited your original post. I'm not a technical person, so I can't tell. Did your change make the original answers not make sense anymore? If they did, I can add the information back in there. I promise I won't disrupt the current edit. Just let me know. 🙂

[korilium]

Hey @Piper Wilson​  yes, I indeed edited my post. I did that to make the question more wholesome: If your question is answered with a question, then your initial question was not clear enough. That is why I edited the post.

I do not know what you want to add but if it makes the conversation more clear, go ahead.

[korilium]

Hey everyone,

I reinstalled everything just like on the website in data thirst.

When I reinstalled it, I got a message that the firewall was blocking certain features of java 8.

As this is a company's pc I could not alter the settings of the firewall. however I went to the IT department and they disabled the company policy protection of my pc. This is basically a Software called Cloudfare. Once this was disabled everything worked fine.

I probably should solve this now within the company.

Thanks for the help!

[Justin09]

In case it helps anyone, I ran into this issue and had to remove the trailing / from the host name. It used to work fine with the trailing / so something must have changed.

[laportina]

It helped, THANK YOU @Justin Allred​ !. Our problems started about the same time as you posted the above. Would not have easily guessed the solution otherwise.