I have a Databricks cluster with 10.4 runtime, when I configure databricks-connect configure I put all the information needed and using the default port 15001, databricks-connect test works.

But changing the port to 443 does not work, I tried to do a pyspark session but it also fails:

: com.databricks.service.SparkServiceConnectionException: Invalid port (443) or invalid token:

The port you specified is either being used already or invalid.
Port: The port that Databricks Connect connects to
  - The default Databricks Connect port is 15001
  - Get current value: spark.conf.get("spark.databricks.service.port")
  - Set via conf: spark.conf.set("spark.databricks.service.port", <your port>")

I added a configuration on my cluster (spark config)

spark.databricks.service.server.enabled true

spark.databricks.service.port 443

But it did not work.

Why doesn't it work? The reason why I want to change to port 443 is simple, my github action workflows is on a self-hosted worker which is behind a firewall that block the port 15001, so I'm trying to find a solution to use a port which is opened.

I tried to set up in `databricks-connect configure` the port 443 + adding it to my spark conf on the driver and on the cluster.