How to Assign User Managed Identity to DBR Cluster so I can use it for quering ADLSv2?

[Filip]

Hi,

I'm trying to figure out if we can switch from Entra ID SPN's to User Assigned Managed Indentities and everything works except I can't figure out how to access the lake files from python notebook.

I've tried with below code and was running it on a cluster as Managed Identity but basically I was getting the same error as when I've tried to run from any differetn cluster:

spark.conf.set("fs.azure.account.auth.type.storageaccount.dfs.core.windows.net", "OAuth")

spark.conf.set("fs.azure.account.oauth.provider.type.storageaccount.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.oauth2.ManagedIdentityTokenProvider")

df = spark.read.csv("abfss://mylakecontainer@storageaccount.dfs.core.windows.net/path/to/csv/file.csv")

df.show()

but I get an error:

IllegalArgumentException: Failed to initialize org.apache.hadoop.fs.azurebfs.oauth2.ManagedIdentityTokenProvider

I could not find any example online where Managed Identity is used to get access to the lake

Then I've decided to try different approach as I've had only reader sole assigned to that Managed Identity so it should be possible to for example print some properties:

from azure.identity import DefaultAzureCredential

from azure.identity import ManagedIdentityCredential

from azure.storage.blob import BlobServiceClient

from azure.mgmt.storage import StorageManagementClient

subscription_id = "111-111-111-111"

resource_group_name = "rmy-rg"

storage_account_name = "mystorage"

account_url = f"https://{storage_account_name}.blob.core.windows.net"

credential = ManagedIdentityCredential()

storage_client = StorageManagementClient(credential, subscription_id)

storage_account = storage_client.storage_accounts.get_properties(

resource_group_name, storage_account_name

)

print("Storage Account Properties:")

print(f"Name: {storage_account.name}")

print(f"Location: {storage_account.location}")

print(f"Kind: {storage_account.kind}")

print(f"SKU: {storage_account.sku.name}")

print(f"Primary Location: {storage_account.primary_location}")

print(f"Status of Primary: {storage_account.status_of_primary}")

And even with my cluster having Access Mode set to "Assigned" and that assignee is the Managed Identity, when I run the above code I get the error:

HttpResponseError: (AuthorizationFailed) The client '1f945563-4de8-44a0-a979-2c4e4540ad4c' with object id '1f945563-4de8-44a0-a979-2c4e4540ad4c' does not have authorization to perform action 'Microsoft.Storage/storageAccounts/read' over scope '/subscriptions/111-111-11/resourceGroups/my-rg/providers/Microsoft.Storage/storageAccounts/mystorage' or the scope is invalid. If access was recently granted, please refresh your credentials.

This client ID 1f945563-4de8-44a0-a979-2c4e4540ad4c is the dbmanagedidentity (enterprise application default one) and not my User Assigned Managed Identity which is added to the workspace as the SPN. Why the job running on that cluser in assigned mode still not using my managed identity? 

In the last  second, I've decided to use

client_id = "my-umi-client-id"

credential = DefaultAzureCredential(managed_identity_client_id=client_id)

But then I get the error:

ClientAuthenticationError: DefaultAzureCredential failed to retrieve a token from the included credentials.

WARNING:azure.identity._credentials.chained:DefaultAzureCredential failed to retrieve a token from the included credentials.

Attempted credentials:

EnvironmentCredential: EnvironmentCredential authentication unavailable. Environment variables are not fully configured.

Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot to troubleshoot this issue.

ManagedIdentityCredential: ManagedIdentityCredential authentication unavailable. The requested identity has not been assigned to this resource. Error: Unexpected response "{'error': 'invalid_request', 'error_description': 'Identity not found'}"

SharedTokenCacheCredential: SharedTokenCacheCredential authentication unavailable. No accounts were found in the cache.

AzureCliCredential: Azure CLI not found on path

AzurePowerShellCredential: PowerShell is not installed

AzureDeveloperCliCredential: Azure Developer CLI could not be found. Please visit https://aka.ms/azure-dev for installation instructions and then,once installed, authenticate to your Azure account using 'azd auth login'.

To mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.

but this managed Identity is assigned to that cluster...

What I'm doing wrong here? 

Azure Databricks Service does not have Identity option and if I go to managed resource group of the workspace, there is just plethora of VM's so I guess this is not the correct way to assign User Managed Identity to the cluster...

[szymon_dybczak]

Hi @Filip ,

It's obsolete way of configuring access to storage account. Nowadays you should use UC and storage credentials and external location to configure access to storage account. 

A storage credential is a securable object representing an Azure managed identity or Microsoft Entra ID service principal. Once a storage credential is created access to it can be granted to principals (users and groups).Storage credentials are primarily used to create external locations, which scope access to a specific storage path

Storage credentials - Azure Databricks - Databricks SQL | Microsoft Learn

[Filip]

Yea I'm aware of that UC is fixing that but I'm not on UC yet and wanted to know if it is even possible to use our own user assigned managed identity and assign it instead of using built-in one as it looks it os not really possible for some reason.

[szymon_dybczak]

Ok, so unfortunately using User Assigned Managed Identity to read/write from ADLS Gen2 inside a notebook is not directly supported. Your best bet is to use regular service principal or switch to unity catalog.

[kuniteru]

Hi,

I can be accessed with the following code.

storageAccountName = "my-storage-account-name"
applicationClientId = "my-umi-client-id"
aadDirectoryId = "my-entra-tenant-id"
containerName = "my-lake-container"

spark.conf.set("fs.azure.account.auth.type", "OAuth")
spark.conf.set("fs.azure.account.oauth.provider.type","org.apache.hadoop.fs.azurebfs.oauth2.MsiTokenProvider")
spark.conf.set("fs.azure.account.oauth2.msi.tenant", aadDirectoryId)
spark.conf.set("fs.azure.account.oauth2.client.id", applicationClientId)

df = spark.read.csv("abfss://"+containerName+"@"+storageAccountName+".dfs.core.windows.net/hello.csv")
df.show()

I too would like to change to UC but can't take the time to do so...

[Ankitbarlota92]

hi. could you tell me what were the cluster settings you used for this one.

[Coffee77]

Best option is to use "External Locations" and "Storage Credentials" under "Unity Catalog". This avoids tons of problems.

If Unity Catalog is not possible, only way to achieve this I was able to verify is:

When you deploy a Databricks workspace, a user-assigned dbManagedIdentity is created in background. Not sure if this only happens in non-VNET injected workspaces or for all setups (pending to check)

Previous dbManagedIdentity is then associated to VMs making up the cluster. So, if you assign proper roles to this managed identity over ADLS, it turns out DefaultAzureCredentials gets this "identity" (checked in access token payload and code on my own) and the code pasted above really works. VMs are using this managed identity to access ADLS 🙂 but (I think) nothing to do with Azure Databricks Connector.

Check here how user-assigned managed identity is assigned to VMs

Kind Regards.

[Coffee77]

Besides, this only works in dedicated clusters, non working on shared ones. Why? No idea at all. Latest case, IMDS (Internal Metadata Service) used by Azure to inject token endpoint inside resources as a unique secure and valid channel to get tokens for managed identities is not created or not accessible. So, we get this error: