topic Re: Databricks-connect invalid shard address in Data Engineering

Databricks-connect invalid shard address

korilium — Thu, 03 Feb 2022 09:55:22 GMT

I want to use databricks inside vscode and I therefore need Databricks-connect

I configure my settings using

databricks-connect configure

as follows:

Databricks Host [https://adb-1409757184094616.16.azuredatabricks.net]

Databricks Token [<my token>]

Cluster ID [1220-124223-ku6xm034]

Org ID [<my organization>]

Port [5001]

following @Kaniz Fatma and @Arjun Kaimaparambil Rajan comments here is some other information that might be relevant:

databricks runtime version (cluster): 9.1 LTS ML
python version in conda (base) environment: 3.8.10
Check certificate issues

These are my Spark config settings of the cluster I want to run in vscode

spark.databricks.cluster.profile serverless
spark.databricks.service.port 8787
spark.sql.legacy.allowNonEmptyLocationInCTAS = true
spark.databricks.service.server.enabled true
spark.databricks.repl.allowedLanguages sql,python,r

when I run

databricks-connect test

in my prompt I get the following error:

ERROR SparkClientManager: Fail to get the SparkClient
 
java.util.concurrent.ExecutionException: com.databricks.service.SparkServiceConnectionException: Invalid shard address: "https://adb-1409757184094616.16.azuredatabricks.net/"

The stacktrace is attached to this question.

I tried to change my shard address to

Databricks Host [https://northeurope.azuredatabricks.net]
Databricks Host [https://westeurope.azuredatabricks.net]

I discussed this with my data engineer at the company and he told me that it could be an issue with the private endpoints that they installed last year. This might be blocking the connection with my remote pc. Anyone any ideas?

Re: Databricks-connect invalid shard address

korilium — Thu, 03 Feb 2022 12:40:15 GMT

what I have configured is the following:

cluster is 9.1 LTS ML and my python version is 3.8.10 in the base conda environment

I did had the server enabled but did not set the port to 8787 in the cluster settings.

I tried setting the port of the cluster to 8787 while keeping the server enabled but it was no good. Still got the same error.

I do not know if I have a 7 layer firewall. Is there a way to see it?

Re: Databricks-connect invalid shard address

korilium — Wed, 09 Feb 2022 14:04:30 GMT

Hi @Arjun Kaimaparambil Rajan , you can find the stacktrace in the attechment of the question. By REST API you mean if I can connect to the databricks workspace via the webbrowser ? (which I can)

Hopefully you can help me with this extra information. already many thanks for the support.

Re: Databricks-connect invalid shard address

Anonymous — Wed, 09 Feb 2022 15:44:20 GMT

@ignace decocq - I noticed that you edited your original post. I'm not a technical person, so I can't tell. Did your change make the original answers not make sense anymore? If they did, I can add the information back in there. I promise I won't disrupt the current edit. Just let me know. 🙂

Re: Databricks-connect invalid shard address

korilium — Wed, 09 Feb 2022 16:32:27 GMT

Hey @Piper Wilson yes, I indeed edited my post. I did that to make the question more wholesome: If your question is answered with a question, then your initial question was not clear enough. That is why I edited the post.

I do not know what you want to add but if it makes the conversation more clear, go ahead.

Re: Databricks-connect invalid shard address

korilium — Mon, 14 Feb 2022 13:59:22 GMT

@Arjun Kaimaparambil Rajan I followed the link you send me and when I run :

I get the following exception:

javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
        at sun.security.ssl.SSLSocketImpl.handleEOF(Unknown Source)
        at sun.security.ssl.SSLSocketImpl.decode(Unknown Source)
        at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source)
        at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
        at sun.security.ssl.SSLSocketImpl.ensureNegotiated(Unknown Source)
        at sun.security.ssl.SSLSocketImpl.access$200(Unknown Source)
        at sun.security.ssl.SSLSocketImpl$AppOutputStream.write(Unknown Source)
        at sun.security.ssl.SSLSocketImpl$AppOutputStream.write(Unknown Source)
        at SSLPoke.main(SSLPoke.java:23)
Caused by: java.io.EOFException: SSL peer shut down incorrectly
        at sun.security.ssl.SSLSocketInputRecord.read(Unknown Source)
        at sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source)
        at sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source)
        at sun.security.ssl.SSLTransport.decode(Unknown Source)
        ... 8 more

I tried solving this exception following this video but to no avail. I am really feeling that I am shooting in the dark for the moment. Hopefully you can help me with that.

Re: Databricks-connect invalid shard address

jose_gonzalez — Tue, 15 Mar 2022 00:10:07 GMT

Hi @ignace decocq

Did @Kaniz Fatma response help you to resolved your question? if yes, please mark it as best response. If not, please let us know and we will continue helping you to find a solution.

Re: Databricks-connect invalid shard address

korilium — Tue, 29 Mar 2022 14:54:21 GMT

Hey everyone,

I reinstalled everything just like on the website in data thirst.

When I reinstalled it, I got a message that the firewall was blocking certain features of java 8.

As this is a company's pc I could not alter the settings of the firewall. however I went to the IT department and they disabled the company policy protection of my pc. This is basically a Software called Cloudfare. Once this was disabled everything worked fine.

I probably should solve this now within the company.

Thanks for the help!

Re: Databricks-connect invalid shard address

Justin09 — Fri, 17 Jun 2022 17:06:50 GMT

In case it helps anyone, I ran into this issue and had to remove the trailing / from the host name. It used to work fine with the trailing / so something must have changed.

Re: Databricks-connect invalid shard address

laportina — Thu, 23 Jun 2022 13:57:05 GMT

It helped, THANK YOU @Justin Allred !. Our problems started about the same time as you posted the above. Would not have easily guessed the solution otherwise.