02-12-2024 12:56 AM
02-12-2024 02:36 AM
Hi @felix_counter, Let’s explore how to authenticate the Databricks provider using a system-managed identity in Azure. System-managed identities (formerly known as Managed Service Identities or MSIs) provide a secure way to authenticate applications and services without managing explicit credentials.
Here are the steps to set up Azure managed identities authentication for Databricks:
Create a User-Assigned Managed Identity:
Assign the Managed Identity to Your Databricks Account and Workspace:
Configure an Azure Virtual Machine (VM):
Install and Configure the Databricks CLI on the Azure VM:
Run Commands with the Databricks CLI:
Here’s how you can set up the managed identity for your existing Web App:
Remember that managed identities for Azure resources are different from Microsoft Entra ID (formerly Azure Active Directory) service principals. Databricks supports both types of authentication, so choose the one that best fits your use case.
By following these steps, you should be able to authenticate the Databricks provider using a system-managed identity. If you encounter any issues, double-check the configuration and ensure that the managed identity has the necessary permissions in both the Databricks workspace and the Azure subscription where it resides.
For more detailed information, refer to the official Azure Databricks documentation on managed identities authentication.
02-12-2024 02:55 AM
Dear @Kaniz_Fatma,
thanks a lot for your response describing the step-by-step guide to authenticate Databricks using a managed identity.
However, to my best understanding this is not what I want to achieve. To recap, my goal is to use a system-assigned (i.e., not a user-assigned) managed identity of a web app to authenticate with the terraform databricks provider (i.e., not the CLI). I would be very grateful if you could provide a similar step-by-step guide for this setup.
02-13-2024 11:42 AM
I furthermore also tried to authenticate using a user-assigned managed identity. In detail, I performed the following steps using Terraform:
The same error ("Identity not found") occurs during the terraform apply of step 5 (token creation). I also tried creating other resources, they all fail with above-stated error message. @alexott, do you have a suggestion?
Thanks a lot for your support!
03-22-2024 03:17 AM
I think I have your answer.
To create a databricks provider to manage your workspace using an SPN, you need to create the provider like this:
provider "databricks" {
alias = "workspace"
host = <your workspace URL>
azure_client_id = <Application ID of the SPN>
azure_client_secret = <Application secret of the SPN>
azure_tenant_id = <Your Azure subscription tenant ID>
}
I store all these credentials as secrets in my Azure KeyVault and call the keyvault to have access to all its secrets. Then I define data fields to retrieve the secret values from my KeyVault and pass them in the databricks provider definition. You probably know you need to use azurerm provider for this. Below is the full block:
data "azurerm_key_vault" "key_vault" {
name = <your keyvault_name>
resource_group_name = <your rg_name>
}
data "azurerm_key_vault_secret" "workspace_url" {
name = "<Workspace-URL>"
key_vault_id = data.azurerm_key_vault.key_vault.id
}
data "azurerm_key_vault_secret" "workspace_admin_spn_app_id" {
name = "<Workspace-ADMINSPN-APPLICATIONID>"
key_vault_id = data.azurerm_key_vault.key_vault.id
}
data "azurerm_key_vault_secret" "workspace_admin_spn_app_secret" {
name = "<Workspace-ADMINSPN-APPLICATIONSECRET>"
key_vault_id = data.azurerm_key_vault.key_vault.id
}
data "azurerm_key_vault_secret" "tenant_id" {
name = "<AZURE-TENANTID>"
key_vault_id = data.azurerm_key_vault.key_vault.id
}
provider "databricks" {
alias = "workspace"
host = data.azurerm_key_vault_secret.workspace_url.value
azure_client_id = data.azurerm_key_vault_secret.workspace_admin_spn_app_id.value
azure_client_secret = data.azurerm_key_vault_secret.workspace_admin_spn_app_secret.value
azure_tenant_id = data.azurerm_key_vault_secret.tenant_id.value
}
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group