Run Terraform from CLI by non-admin users

[Andrei_Radulesc]

We have a team of multiple people, and they need to do Databricks Terraform config from the CLI. We'd like to do this without having all people be account admins in Databricks.

What is the best way to do that?

Right now, we have Terraform set up so that you must be admin of Databricks account to run Terraform. The account provider is set up with user/password:

provider "databricks" {

 alias     = "mws"

 host      = "https://accounts.cloud.databricks.com"

 account_id = var.databricks_account_id

 username  = var.databricks_account_username

 password  = var.databricks_account_password

}

Workspace providers are set up with workspace tokens, and you don't have to be workspace admin (just need a workspace admin token):

provider "databricks" {

 alias = "dev_workspace"

 host = module.dev_workspace.workspace_url

 token = module.dev_workspace.databricks_token

}

Would it be possible to configure the Databricks account Terraform provider, say, with some sort of token with admin privileges, or some sort of Service Principal with admin privileges - so we don't need to be admin user to run Terraform?

[Anonymous · Accepted solution]

@Andrei Radulescu-Banu​ :

Yes, it is possible to configure the Databricks account Terraform provider with a Service Principal that has admin privileges, so you don't need to be an admin user to run Terraform.

Here are the steps to set this up:

1. Create a Service Principal in Azure AD with the appropriate permissions. The Service Principal should have the following permissions:

• Azure Databricks Service role assignment at the resource group or subscription level
  • Contributor or Owner role assignment at the resource group or subscription level (this is required to create and manage resources in Azure)

1. Generate a client secret for the Service Principal.
2. Set up the Databricks provider in your Terraform configuration to use the Service Principal credentials:

provider "databricks" {
  alias        = "mws"
  host         = "https://accounts.cloud.databricks.com"
  account_id   = var.databricks_account_id
  client_id    = var.service_principal_client_id
  client_secret = var.service_principal_client_secret
}

1. Add the necessary variables to your Terraform configuration to store the Service Principal credentials securely.

With this setup, you can grant access to the Service Principal to perform administrative tasks without needing to give admin access to all users in your Databricks account. This provides a more secure and controlled way to manage your infrastructure.

[Anonymous]

Hi @Andrei Radulescu-Banu​ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!

[Andrei_Radulesc]

Thank you Suteja. It sort-of works, but I am still encountering authentication problems. I have updated to Databricks terraform provider 1.13.0 for this. Here is my account provider:

provider "databricks" {

 alias     = "mws"

 host      = "https://accounts.cloud.databricks.com"

 account_id = var.databricks_account_id // This is needed e.g. by databricks_groups

# OAuth service principal config

 client_id  = var.databricks_account_service_principal_uuid

 client_secret = var.databricks_account_service_principal_secret

# Old user/pasword config

#  username  = var.databricks_account_username

#  password  = var.databricks_account_password

}

Here is my workspace provider:

provider "databricks" {

 alias = "dev_workspace"

 host = ... # Pass the workspace URL here

# OAuth service principal config

 client_id = var.databricks_account_service_principal_uuid

 client_secret = var.databricks_account_service_principal_secret

# Old token-based config

 # token = module.dev_workspace.databricks_token

}

With this setup, I was able to get it to work, at least initially. Then, I started getting permission errors (see below), strangely, after it all worked for a full morning. So I had to back off the changes and go back to user/password config at the account level, and token config at the workspace level.

In our case, the workspaces had been initially created with user/password, and are used in production. So I can't recreate these workspaces from scratch. I had, instead, to move the Terraform config for these workspaces to use Service Principal.

The service principal was manually created, in the UI, at the account level. And the service principal is given 'admin' permissions at the account level.

I can't remember how I configured the Service Principal at the workspace level. Looking at the GUI, the Service Principal shows up both at the Account and at the Workspace level - so it is associated to the Workspace. Can't remember how I did that. I think it was automatic, because the Service Principal was manually configured to be "admin" at the account level.

After making these changes in the UI, and in Terraform, I started re-applying the old terraform config. It gave me permissions error in Unity Catalog related entitied - because those had been created with the old user/password and old token config, so they ended up being owned by myself.

Those were resolved by going in the UI, and manually changing ownership of those entities to the Service Principal.

So, all was good for a morning. Then, in the afternoon, I started getting the errors below:

2023-03-24T13:23:49.452-0400 [INFO] provider.terraform-provider-databricks_v1.13.0: Workspace is now running: timestamp=2023-03-24T13:23:49.452-0400

2023-03-24T13:23:49.605-0400 [DEBUG] provider.terraform-provider-databricks_v1.13.0: Generating Databricks OAuth token for Service Principal (***): timestamp=2023-03-24T13:23:49.605-0400

2023-03-24T13:23:49.646-0400 [DEBUG] provider.terraform-provider-databricks_v1.13.0: Generating Databricks OAuth token for Service Principal (***): timestamp=2023-03-24T13:23:49.646-0400

2023-03-24T13:23:49.711-0400 [DEBUG] provider.terraform-provider-databricks_v1.13.0: Generating Databricks OAuth token for Service Principal (***): timestamp=2023-03-24T13:23:49.711-0400

2023-03-24T13:23:49.990-0400 [DEBUG] provider.terraform-provider-databricks_v1.13.0: GET /api/2.0/token/list

< HTTP/2.0 401 Unauthorized

[non-JSON document of 274 bytes]: timestamp=2023-03-24T13:23:49.990-0400

2023-03-24T13:23:49.990-0400 [INFO] provider.terraform-provider-databricks_v1.13.0: workspace https://***.cloud.databricks.com is not yet reachable: Authentication failed: timestamp=2023-03-24T13:23:49.990-0400

2023-03-24T13:23:50.188-0400 [DEBUG] provider.terraform-provider-databricks_v1.13.0: GET /api/2.0/token/list

< HTTP/2.0 401 Unauthorized

[non-JSON document of 274 bytes]: timestamp=2023-03-24T13:23:50.188-0400

2023-03-24T13:23:50.188-0400 [INFO] provider.terraform-provider-databricks_v1.13.0: workspace https://***.cloud.databricks.com is not yet reachable: Authentication failed: timestamp=2023-03-24T13:23:50.188-0400

2023-03-24T13:23:50.284-0400 [DEBUG] provider.terraform-provider-databricks_v1.13.0: GET /api/2.0/token/list

< HTTP/2.0 401 Unauthorized

[non-JSON document of 274 bytes]: timestamp=2023-03-24T13:23:50.284-0400

2023-03-24T13:23:50.284-0400 [INFO] provider.terraform-provider-databricks_v1.13.0: workspace https://***.cloud.databricks.com is not yet reachable: Authentication failed: timestamp=2023-03-24T13:23:50.284-0400

I ***'d the links and the IDs in the output. It just got in a loop with failed authentication, and I was never able to shake that out.

The only solution was to back off from using the Service Principal, and go back to user/password and token authentication. Luckily, I was able to back off without losing the production workspaces.

[Andrei_Radulesc]

I think the Service Principal definitely works for workspaces created from scratch with the Service Principal. I've tested that in my test environment, and have not encountered these kinds of authentication problem.

The issue I am seeing is with workspaces created with user/password and token based authentication. Something, deep in those workspaces, still has permissions set up in such a way that Service Principal based configuration gives authentication errors.