Azure IMDS is not accesbile selecting shared compute policy

[sharukh_lodhi]

Hi, Databricks community,
I recently encountered an issue while using the 'azure.identity' Python library on a cluster set to the personal compute policy in Databricks. In this case, Databricks successfully returns the Azure Databricks managed user identity. However, when I change the cluster policy to shared compute, this functionality stops working.

Here's the code I've been using:

from azure.identity import DefaultAzureCredential
credential = DefaultAzureCredential() 

Please note that I've also attached a screenshot of the error that occurs only when selecting the shared compute cluster policy.

[tyler-xorbix]

This appears to be a long-standing issue with Azure and Databricks. I found this stack overflow post from a few months back discussing the same issue with no outcome. ManagedIdentityCredential in Shared Compute - Databricks - Stack Overflow. I suspect it is either a glitch or a security override since a shared cluster can be used by multiple users. Hopefully someone from Databricks end can look into this issue further.