I tried with creating a new cluster, for 10.4, but that didn't get my anywhere either. The steps I followed where:

```

$ databricks clusters create --json-file cluster.json

```

Where `cluster.json` looks like

```

{

  "cluster_name": "test50",

  "spark_version": "10.4.x-scala2.12",

  "spark_conf": {

    "spark.databricks.service.client.enabled": true,

    "spark.databricks.service.server.enabled": true,

    "spark.speculation": true,

    "spark.sql.session.timeZone": "UTC"

  },

  "spark_env_vars": {

    "PYSPARK_PYTHON": "/databricks/python3/bin/python3"

  },

  "node_type_id": "i3.xlarge",

  "autoscale": {

    "min_workers": 1,

    "max_workers": 8

  },

  "autotermination_minutes": 10,

  "aws_attributes": {

    "first_on_demand": 0,

    "availability": "SPOT_WITH_FALLBACK",

    "zone_id": "eu-west-1b",

    "spot_bid_price_percent": 100

  },

  "enable_elastic_disk": false,

  "data_security_mode": "SINGLE_USER",

  "single_user_name": "****"

}

```

And then

```

$ python3 -m venv ~/databricks12

$ . ~/databricks12/bin/activate

$ pip install --upgrade pip

$ pip install --upgrade setuptools

$ pip install databricks-connect==10.4.18

$ databricks-connect test

```

And the result is as before

```

23/02/09 10:22:14 ERROR SparkServiceRPCClient: Failed to sync with the spark cluster. This could be a intermittent issue, please check your cluster's state and retry.

com.databricks.service.SparkServiceConnectionException: Invalid token

To connect to a Databricks cluster, you must specify an API token.

API Token: The API token used to confirm your identity to Databricks

 - Learn more about API tokens here: https://docs.databricks.com/api/latest/authentication.html#generate-a-token

 - Get current value: spark.conf.get("spark.databricks.service.token")

 - Set via conf: spark.conf.set("spark.databricks.service.token", <your API token>)

 - Set via environment variable: export DATABRICKS_API_TOKEN=<your API token>

```