Service Principal Authentication / Terraform

[naumanmunir]

Hello Databricks Community,

I'm encountering an issue when trying to apply my Terraform configuration to create a Databricks MWS network on GCP. The terraform apply command fails with the following error:

Error: cannot create mws networks: failed during request visitor: default auth: cannot configure default credentials, please check https://docs.databricks.com/en/dev-tools/auth.html#databricks-client-unified-authentication to configure credentials for your preferred authentication method. Config: host=https://accounts.gcp.databricks.com, account_id=<accountid>, client_id=<clientid>, client_secret=***

  with databricks_mws_networks.databricks_network["dbx-workspace"],
  on main.tf line 1, in resource "databricks_mws_networks" "databricks_network":
   1: resource "databricks_mws_networks" "databricks_network" {

For context, the service principal I'm attempting to use for this Terraform deployment was created directly through the Databricks Account Console UI. I have the client_id (Application ID) of this service principal. The client id and the client secret generated are saved in gcp secret manager

Here is my versions.tf file:

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 6.0"
    }
    google-beta = {
      source  = "hashicorp/google-beta"
      version = "~> 6.0"
    }
    databricks = {
      source  = "databricks/databricks"
      version = "~> 1.74.0"
    }
  }
  required_version = ">= 1.6.3"

}

Here is the provider.tf file

provider "databricks" {
  alias                  = "account"
  host                   = "https://accounts.gcp.databricks.com"
  account_id             = var.databricks_account_id
  client_id = data.google_secret_manager_secret_version.databricks_client_id.secret_data
  client_secret = data.google_secret_manager_secret_version.databricks_client_secret.secret_data
  auth_type = "google-id"

[mnorland]

Databricks account-level APIs can only be called by account owners and account admins and can only be authenticated using Google-issued OIDC tokens.

In Terraform 0.13 and later, data resources have the same dependency resolution behavior as defined for managed resources. Most data resources make an API call to a workspace. If a workspace doesn't exist yet, default auth: cannot configure default credentials error is raised. To work around this issue and guarantee proper lazy authentication with data resources, you should add depends_on = [databricks_mws_workspaces.this] to the body. This issue doesn't occur if a workspace is created in one module and resources within the workspace are created in another. 

Please refer to this URL for more information:
https://github.com/databricks/terraform-provider-databricks/blob/main/docs/guides/gcp-workspace.md