I want to use databricks inside vscode and I therefore need Databricks-connect

I configure my settings using

databricks-connect configure

as follows:

Databricks Host [https://adb-1409757184094616.16.azuredatabricks.net]

Databricks Token [<my token>]

Cluster ID [1220-124223-ku6xm034]

Org ID [<my organization>]

Port [5001]

following @Kaniz Fatma​ and @Arjun Kaimaparambil Rajan​ comments here is some other information that might be relevant:

• databricks runtime version (cluster): 9.1 LTS ML
• python version in conda (base) environment: 3.8.10
• Check certificate issues

These are my Spark config settings of the cluster I want to run in vscode

spark.databricks.cluster.profile serverless
spark.databricks.service.port 8787
spark.sql.legacy.allowNonEmptyLocationInCTAS = true
spark.databricks.service.server.enabled true
spark.databricks.repl.allowedLanguages sql,python,r

when I run

databricks-connect test

in my prompt I get the following error:

ERROR SparkClientManager: Fail to get the SparkClient
 
java.util.concurrent.ExecutionException: com.databricks.service.SparkServiceConnectionException: Invalid shard address: "https://adb-1409757184094616.16.azuredatabricks.net/"

The stacktrace is attached to this question.

I tried to change my shard address to

• Databricks Host [https://northeurope.azuredatabricks.net]
• Databricks Host [https://westeurope.azuredatabricks.net]

I discussed this with my data engineer at the company and he told me that it could be an issue with the private endpoints that they installed last year. This might be blocking the connection with my remote pc. Anyone any ideas?