Thank you Suteja. It sort-of works, but I am still encountering authentication problems. I have updated to Databricks terraform provider 1.13.0 for this. Here is my account provider:

provider "databricks" {

 alias     = "mws"

 host      = "https://accounts.cloud.databricks.com"

 account_id = var.databricks_account_id // This is needed e.g. by databricks_groups

# OAuth service principal config

 client_id  = var.databricks_account_service_principal_uuid

 client_secret = var.databricks_account_service_principal_secret

# Old user/pasword config

#  username  = var.databricks_account_username

#  password  = var.databricks_account_password

}

Here is my workspace provider:

provider "databricks" {

 alias = "dev_workspace"

 host = ... # Pass the workspace URL here

# OAuth service principal config

 client_id = var.databricks_account_service_principal_uuid

 client_secret = var.databricks_account_service_principal_secret

# Old token-based config

 # token = module.dev_workspace.databricks_token

}

With this setup, I was able to get it to work, at least initially. Then, I started getting permission errors (see below), strangely, after it all worked for a full morning. So I had to back off the changes and go back to user/password config at the account level, and token config at the workspace level.

In our case, the workspaces had been initially created with user/password, and are used in production. So I can't recreate these workspaces from scratch. I had, instead, to move the Terraform config for these workspaces to use Service Principal.

The service principal was manually created, in the UI, at the account level. And the service principal is given 'admin' permissions at the account level.

I can't remember how I configured the Service Principal at the workspace level. Looking at the GUI, the Service Principal shows up both at the Account and at the Workspace level - so it is associated to the Workspace. Can't remember how I did that. I think it was automatic, because the Service Principal was manually configured to be "admin" at the account level.

After making these changes in the UI, and in Terraform, I started re-applying the old terraform config. It gave me permissions error in Unity Catalog related entitied - because those had been created with the old user/password and old token config, so they ended up being owned by myself.

Those were resolved by going in the UI, and manually changing ownership of those entities to the Service Principal.

So, all was good for a morning. Then, in the afternoon, I started getting the errors below:

2023-03-24T13:23:49.452-0400 [INFO] provider.terraform-provider-databricks_v1.13.0: Workspace is now running: timestamp=2023-03-24T13:23:49.452-0400

2023-03-24T13:23:49.605-0400 [DEBUG] provider.terraform-provider-databricks_v1.13.0: Generating Databricks OAuth token for Service Principal (***): timestamp=2023-03-24T13:23:49.605-0400

2023-03-24T13:23:49.646-0400 [DEBUG] provider.terraform-provider-databricks_v1.13.0: Generating Databricks OAuth token for Service Principal (***): timestamp=2023-03-24T13:23:49.646-0400

2023-03-24T13:23:49.711-0400 [DEBUG] provider.terraform-provider-databricks_v1.13.0: Generating Databricks OAuth token for Service Principal (***): timestamp=2023-03-24T13:23:49.711-0400

2023-03-24T13:23:49.990-0400 [DEBUG] provider.terraform-provider-databricks_v1.13.0: GET /api/2.0/token/list

< HTTP/2.0 401 Unauthorized

[non-JSON document of 274 bytes]: timestamp=2023-03-24T13:23:49.990-0400

2023-03-24T13:23:49.990-0400 [INFO] provider.terraform-provider-databricks_v1.13.0: workspace https://***.cloud.databricks.com is not yet reachable: Authentication failed: timestamp=2023-03-24T13:23:49.990-0400

2023-03-24T13:23:50.188-0400 [DEBUG] provider.terraform-provider-databricks_v1.13.0: GET /api/2.0/token/list

< HTTP/2.0 401 Unauthorized

[non-JSON document of 274 bytes]: timestamp=2023-03-24T13:23:50.188-0400

2023-03-24T13:23:50.188-0400 [INFO] provider.terraform-provider-databricks_v1.13.0: workspace https://***.cloud.databricks.com is not yet reachable: Authentication failed: timestamp=2023-03-24T13:23:50.188-0400

2023-03-24T13:23:50.284-0400 [DEBUG] provider.terraform-provider-databricks_v1.13.0: GET /api/2.0/token/list

< HTTP/2.0 401 Unauthorized

[non-JSON document of 274 bytes]: timestamp=2023-03-24T13:23:50.284-0400

2023-03-24T13:23:50.284-0400 [INFO] provider.terraform-provider-databricks_v1.13.0: workspace https://***.cloud.databricks.com is not yet reachable: Authentication failed: timestamp=2023-03-24T13:23:50.284-0400

I ***'d the links and the IDs in the output. It just got in a loop with failed authentication, and I was never able to shake that out.

The only solution was to back off from using the Service Principal, and go back to user/password and token authentication. Luckily, I was able to back off without losing the production workspaces.