topic Invalid shard address in Data Engineering

Invalid shard address

Anonymous — Tue, 14 Jun 2022 12:23:26 GMT

I'm running pyspark through databricks-connect and getting an error saying

```

ERROR SparkClientManager: Fail to get the SparkClient

java.util.concurrent.ExecutionException: com.databricks.service.SparkServiceConnectionException: Invalid shard address:

```

followed by my workspace address "https://dbc-xxxxxxxx-xxxx.cloud.databricks.com"

I didn't make any modifications or updates to my python or databricks-connect installation. How can I resolve this issue?

I have looked at a similar thread https://community.databricks.com/s/topic/0TO3f000000CmP9GAK/invalid-shard-address but that discussion did not provide any useful solution to solve my issue.

Prabakar — Thu, 16 Jun 2022 06:43:36 GMT

hi @Marco Wong was this working before and failing now?

Are you behind a VPN or firewall? If so can you check by disabling it?

enable traces at wireshark and collected dump to check if there is traffic going to workspace?

Check if you can get curl working with Databricks rest api for anything (workspace, cluster manager.. etc)?

Anonymous — Thu, 16 Jun 2022 08:09:55 GMT

Yes it was working before and now failing. I'm not behind a VPN/firewall. I'll try using wireshark and curl.

Anonymous — Thu, 16 Jun 2022 09:13:17 GMT

curl works with the databricks api but databricks-connect still doesn't work with the invalid shard address error

jose_gonzalez — Tue, 30 Aug 2022 16:33:28 GMT

Hi,

Just a friendly follow-up. Did you have time to try Prabakar's response? let us know if you still need help or not.