cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Databricks-connect invalid shard address

korilium
New Contributor III

I want to use databricks inside vscode and I therefore need Databricks-connect

I configure my settings using

databricks-connect configure

as follows:

Databricks Host [https://adb-1409757184094616.16.azuredatabricks.net]

Databricks Token [<my token>]

Cluster ID [1220-124223-ku6xm034]

Org ID [<my organization>]

Port [5001]

following @Kaniz Fatma​ and @Arjun Kaimaparambil Rajan​ comments here is some other information that might be relevant:

  • databricks runtime version (cluster): 9.1 LTS ML
  • python version in conda (base) environment: 3.8.10
  • Check certificate issues

These are my Spark config settings of the cluster I want to run in vscode

spark.databricks.cluster.profile serverless
spark.databricks.service.port 8787
spark.sql.legacy.allowNonEmptyLocationInCTAS = true
spark.databricks.service.server.enabled true
spark.databricks.repl.allowedLanguages sql,python,r

when I run

databricks-connect test

in my prompt I get the following error:

ERROR SparkClientManager: Fail to get the SparkClient
 
java.util.concurrent.ExecutionException: com.databricks.service.SparkServiceConnectionException: Invalid shard address: "https://adb-1409757184094616.16.azuredatabricks.net/"

The stacktrace is attached to this question.

I tried to change my shard address to

I discussed this with my data engineer at the company and he told me that it could be an issue with the private endpoints that they installed last year. This might be blocking the connection with my remote pc. Anyone any ideas?

1 ACCEPTED SOLUTION

Accepted Solutions

Hi @ignace decocq​ , Please go through the steps provided in the link. Thank you.

View solution in original post

11 REPLIES 11

Kaniz_Fatma
Community Manager
Community Manager

Hi @ignace decocq​ , You need to check the version of all the tools.

Are you behind a proxy or a network that may have a layer 7 firewall?

Have you set:

spark.databricks.service.server.enabled true
 
spark.databricks.service.port 8787
 

korilium
New Contributor III

what I have configured is the following:

cluster is 9.1 LTS ML and my python version is 3.8.10 in the base conda environment

I did had the server enabled but did not set the port to 8787 in the cluster settings.

I tried setting the port of the cluster to 8787 while keeping the server enabled but it was no good. Still got the same error.

I do not know if I have a 7 layer firewall. Is there a way to see it?

korilium
New Contributor III

Hi @Arjun Kaimaparambil Rajan​ , you can find the stacktrace in the attechment of the question. By REST API you mean if I can connect to the databricks workspace via the webbrowser ? (which I can)

Hopefully you can help me with this extra information. already many thanks for the support.

korilium
New Contributor III

@Arjun Kaimaparambil Rajan​ I followed the link you send me and when I run :

I get the following exception:

javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
        at sun.security.ssl.SSLSocketImpl.handleEOF(Unknown Source)
        at sun.security.ssl.SSLSocketImpl.decode(Unknown Source)
        at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source)
        at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
        at sun.security.ssl.SSLSocketImpl.ensureNegotiated(Unknown Source)
        at sun.security.ssl.SSLSocketImpl.access$200(Unknown Source)
        at sun.security.ssl.SSLSocketImpl$AppOutputStream.write(Unknown Source)
        at sun.security.ssl.SSLSocketImpl$AppOutputStream.write(Unknown Source)
        at SSLPoke.main(SSLPoke.java:23)
Caused by: java.io.EOFException: SSL peer shut down incorrectly
        at sun.security.ssl.SSLSocketInputRecord.read(Unknown Source)
        at sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source)
        at sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source)
        at sun.security.ssl.SSLTransport.decode(Unknown Source)
        ... 8 more

I tried solving this exception following this video but to no avail. I am really feeling that I am shooting in the dark for the moment. Hopefully you can help me with that.

Hi @ignace decocq​ , Please go through the steps provided in the link. Thank you.

Hi @ignace decocq​ 

Did @Kaniz Fatma​ response help you to resolved your question? if yes, please mark it as best response. If not, please let us know and we will continue helping you to find a solution.

Anonymous
Not applicable

@ignace decocq​ - I noticed that you edited your original post. I'm not a technical person, so I can't tell. Did your change make the original answers not make sense anymore? If they did, I can add the information back in there. I promise I won't disrupt the current edit. Just let me know. 🙂

korilium
New Contributor III

Hey @Piper Wilson​  yes, I indeed edited my post. I did that to make the question more wholesome: If your question is answered with a question, then your initial question was not clear enough. That is why I edited the post.

I do not know what you want to add but if it makes the conversation more clear, go ahead.

korilium
New Contributor III

Hey everyone,

I reinstalled everything just like on the website in data thirst.

When I reinstalled it, I got a message that the firewall was blocking certain features of java 8.

As this is a company's pc I could not alter the settings of the firewall. however I went to the IT department and they disabled the company policy protection of my pc. This is basically a Software called Cloudfare. Once this was disabled everything worked fine.

I probably should solve this now within the company.

Thanks for the help!

Justin09
New Contributor II

In case it helps anyone, I ran into this issue and had to remove the trailing / from the host name. It used to work fine with the trailing / so something must have changed.

laportina
New Contributor II

It helped, THANK YOU @Justin Allred​ !. Our problems started about the same time as you posted the above. Would not have easily guessed the solution otherwise.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!