Azure IMDS is not accesbile selecting shared compute policy

[sharukh_lodhi]

Hi, Databricks community,
I recently encountered an issue while using the 'azure.identity' Python library on a cluster set to the personal compute policy in Databricks. In this case, Databricks successfully returns the Azure Databricks managed user identity. However, when I change the cluster policy to shared compute, this functionality stops working.

Here's the code I've been using:

from azure.identity import DefaultAzureCredential
credential = DefaultAzureCredential() 

Please note that I've also attached a screenshot of the error that occurs only when selecting the shared compute cluster policy.

[tyler-xorbix]

This appears to be a long-standing issue with Azure and Databricks. I found this stack overflow post from a few months back discussing the same issue with no outcome. ManagedIdentityCredential in Shared Compute - Databricks - Stack Overflow. I suspect it is either a glitch or a security override since a shared cluster can be used by multiple users. Hopefully someone from Databricks end can look into this issue further.

[Kaniz_Fatma · Accepted solution]

Hi @sharukh_lodhi and @tyler-xorbix, From the error it looks like azure.identity library is trying to connect to the azure IMDS endpoint which is (169.254.169.254) Ip address endpoint and this looks to be a known limitation with shared access mode which is mentioned below document:
https://learn.microsoft.com/en-us/azure/databricks/compute/access-mode-limitations#network-and-file-....

[Kaniz_Fatma]

Hi @sharukh_lodhi,Thanks for reaching out! Please review the responses and let us know which best addresses your question. Your feedback is valuable to us and the community. If the response resolves your issue, kindly mark it as the accepted solution. This will help close the thread and assist others with similar queries. We appreciate your participation and are here if you need further assistance!