topic Authenticating to Accounts Console Using Client ID and Secret via Terraform and Databricks CLI in Data Governance

Authenticating to Accounts Console Using Client ID and Secret via Terraform and Databricks CLI

jv_v — Tue, 11 Jun 2024 14:35:19 GMT

I am currently working on a project where I need to authenticate to the Databricks accounts console from Terraform using a client ID and client secret. Here is the relevant portion of my Terraform configuration:

// Provider for Databricks account
provider "databricks" {
alias = "azure_account"
host = "https://accounts.azuredatabricks.net"
account_id = var.account_id
azure_client_id = var.client_id
azure_client_secret = var.client_secret
}
I execute all the Terraform tasks from a shell script. Within this script, we use a Databricks CLI command to check whether the workspace is assigned to a metastore:

sh
databricks auth login --host https://accounts.azuredatabricks.net --account-id "$account_id"

assignment=$(databricks account metastore-assignments get "$workspace_id" 2>/dev/null)

However, I am unable to authenticate since the command attempts to authenticate using a client ID and client secret.

Is there a way to authenticate using the client ID and client secret directly with the Databricks CLI? If not, can you suggest any alternative methods to achieve this authentication?

Re: Authenticating to Accounts Console Using Client ID and Secret via Terraform and Databricks CLI

JianWu — Tue, 11 Jun 2024 20:42:48 GMT

Hi,

You can follow instructions in the following doc to configure OAuth machine-to-machine authentication using Service Principal.

https://docs.databricks.com/en/dev-tools/cli/authentication.html#oauth-machine-to-machine-m2m-authentication

Once it is configured properly, you should be able to run Databricks CLI, then run following command to confirm metastore assignment for a given workspace:

databricks account metastore-assignments get <<WORKSPACE_ID>>

Re: Authenticating to Accounts Console Using Client ID and Secret via Terraform and Databricks CLI

jv_v — Tue, 18 Jun 2024 13:33:04 GMT

I am attempting to complete OAuth M2M (Machine-to-Machine) authentication as advised, but I keep encountering the following error:

{
    "error": "invalid_request",
    "error_id": "*****************",
    "error_description": "Invalid request"
}

Here is a summary of my current setup and the steps I have taken:

Script used to generate the tocken

export CLIENT_ID=<client id>
export CLIENT_SECRET=clent secret

curl --request POST \
--url 'https://accounts.azuredatabricks.net/oidc/accounts/<accoutnt id>/v1/token \'
--user "$CLIENT_ID:$CLIENT_SECRET" \
--data 'grant_type=client_credentials&scope=all-apis'

Re: Authenticating to Accounts Console Using Client ID and Secret via Terraform and Databricks CLI

JianWu — Tue, 18 Jun 2024 16:15:35 GMT

First, run following commands in shell, please replace placeholder according to your environment:

export CLIENT_ID=<client id> export CLIENT_SECRET=<client secret> export TOKEN_EP=https://accounts.cloud.databricks.com/oidc/accounts/<databricks account id>/v1/token

then run this command to generate token:

curl --request POST --url $TOKEN_EP --user "$CLIENT_ID:$CLIENT_SECRET" --data 'grant_type=client_credentials&scope=all-apis'

Please make sure you follow following instructions to create client id/secret for your service principal, it should be created at account level instead of workspace level.

https://docs.databricks.com/en/dev-tools/auth/oauth-m2m.html#step-3-create-an-oauth-secret-for-a-service-principal