Azure IMDS is not accesbile selecting shared compute policy

[sharukh_lodhi]

Hi, Databricks community,
I recently encountered an issue while using the 'azure.identity' Python library on a cluster set to the personal compute policy in Databricks. In this case, Databricks successfully returns the Azure Databricks managed user identity. However, when I change the cluster policy to shared compute, this functionality stops working.

Here's the code I've been using:

from azure.identity import DefaultAzureCredential
credential = DefaultAzureCredential() 

Please note that I've also attached a screenshot of the error that occurs only when selecting the shared compute cluster policy.

[tyler-xorbix]

This appears to be a long-standing issue with Azure and Databricks. I found this stack overflow post from a few months back discussing the same issue with no outcome. ManagedIdentityCredential in Shared Compute - Databricks - Stack Overflow. I suspect it is either a glitch or a security override since a shared cluster can be used by multiple users. Hopefully someone from Databricks end can look into this issue further.

[Retired_mod]

Hi @sharukh_lodhi and @tyler-xorbix, From the error it looks like azure.identity library is trying to connect to the azure IMDS endpoint which is (169.254.169.254) Ip address endpoint and this looks to be a known limitation with shared access mode which is mentioned below document:
https://learn.microsoft.com/en-us/azure/databricks/compute/access-mode-limitations#network-and-file-....

[Retired_mod]

Hi @sharukh_lodhi,Thanks for reaching out! Please review the responses and let us know which best addresses your question. Your feedback is valuable to us and the community. If the response resolves your issue, kindly mark it as the accepted solution. This will help close the thread and assist others with similar queries. We appreciate your participation and are here if you need further assistance!

[daisy08]

I'm having a similar problem, my aim is from an Azure DataBricks notebook to invoke an AzureDataDactory pipeline I created an Access Connector for Azure Databricks to which I gave Data Factory Contributor permissions. Using these lines python

from azure.identity import ManagedIdentityCredential

credential = ManagedIdentityCredential(client_id="63d0a80e-de48-4dfe-b4ce-a1b2cb90d494")
credential.get_token("https://management.azure.com/.default")

#client_id value is yhe object id of Access Connector for Azure Databricks

I'have error 

WARNING:azure.identity._internal.msal_managed_identity_client:ImdsCredential.get_token failed: ManagedIdentityCredential authentication unavailable, no response from the IMDS endpoint. invalid_request
WARNING:azure.identity._internal.decorators:ManagedIdentityCredential.get_token failed: ManagedIdentityCredential authentication unavailable, no response from the IMDS endpoint. invalid_request

any suggestion?

[Malthe]

How does this work with serverless (for example with DLT pipelines) which runs in standard access mode:

Serverless compute is based on Databricks standard access mode compute architecture (formerly called shared access mode).

To my understanding, from a logical perspective, this mode means that compute is shared with the workspace; as such, it should be possible to attach a managed identity to it.

But testing confirms that the IMDS endpoint is unavailable here as well.