- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-06-2023 05:27 AM
As for the problem above it seems to have gone away. While I'm not sure, it felt a bit like I didn't do anything different. Buy instead I'm faced with a much more mundane situation.
Again, I'm here trying to make `databricks-connect` work.
I simply do
```
$ python3 -m venv ~/databricks11
$ . ~/databricks11/bin/activate
$ pip install --upgrade pip
$ pip install --upgrade setuptools
$ pip install databricks-connect==11.3.0b0
$ databricks-connect configure
$ databricks-connect test
```
My `.databricks-connect` looks like
```
{
"host": "https://dbc-****.cloud.databricks.com",
"token": "dapi****",
"cluster_id": "0110-****,
"port": "15001"
}
```
I also have some environment variables, just in case
```
DATABRICKS_ADDRESS=https://dbc-****.cloud.databricks.com
DATABRICKS_API_TOKEN=dapi****
DATABRICKS_CLUSTER_ID=0110-****
DATABRICKS_PORT=15001
```
But I get an error
```
23/02/03 11:47:17 ERROR SparkClientManager: Fail to get the SparkClient
java.util.concurrent.ExecutionException: com.databricks.service.SparkServiceConnectionException: Invalid token
To connect to a Databricks cluster, you must specify an API token.
API Token: The API token used to confirm your identity to Databricks
- Learn more about API tokens here: https://docs.databricks.com/api/latest/authentication.html#generate-a-token
- Get current value: spark.conf.get("spark.databricks.service.token")
- Set via conf: spark.conf.set("spark.databricks.service.token", <your API token>)
- Set via environment variable: export DATABRICKS_API_TOKEN=<your API token>
```