cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Azure IMDS is not accesbile selecting shared compute policy

sharukh_lodhi
New Contributor III

Hi, Databricks community,
I recently encountered an issue while using the 'azure.identity' Python library on a cluster set to the personal compute policy in Databricks. In this case, Databricks successfully returns the Azure Databricks managed user identity. However, when I change the cluster policy to shared compute, this functionality stops working.

Here's the code I've been using:

 

from azure.identity import DefaultAzureCredential
credential = DefaultAzureCredential() 

 

Please note that I've also attached a screenshot of the error that occurs only when selecting the shared compute cluster policy.

image.png

2 REPLIES 2

tyler-xorbix
New Contributor III

This appears to be a long-standing issue with Azure and Databricks. I found this stack overflow post from a few months back discussing the same issue with no outcome. ManagedIdentityCredential in Shared Compute - Databricks - Stack Overflow. I suspect it is either a glitch or a security override since a shared cluster can be used by multiple users. Hopefully someone from Databricks end can look into this issue further.

Retired_mod
Esteemed Contributor III

Hi @sharukh_lodhi,Thanks for reaching out! Please review the responses and let us know which best addresses your question. Your feedback is valuable to us and the community. If the response resolves your issue, kindly mark it as the accepted solution. This will help close the thread and assist others with similar queries. We appreciate your participation and are here if you need further assistance!

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group