Azure IMDS is not accesbile selecting shared compute policy
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-19-2024 01:58 AM - edited 08-19-2024 01:59 AM
Hi, Databricks community,
I recently encountered an issue while using the 'azure.identity' Python library on a cluster set to the personal compute policy in Databricks. In this case, Databricks successfully returns the Azure Databricks managed user identity. However, when I change the cluster policy to shared compute, this functionality stops working.
Here's the code I've been using:
from azure.identity import DefaultAzureCredential
credential = DefaultAzureCredential()
Please note that I've also attached a screenshot of the error that occurs only when selecting the shared compute cluster policy.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-19-2024 11:22 AM
This appears to be a long-standing issue with Azure and Databricks. I found this stack overflow post from a few months back discussing the same issue with no outcome. ManagedIdentityCredential in Shared Compute - Databricks - Stack Overflow. I suspect it is either a glitch or a security override since a shared cluster can be used by multiple users. Hopefully someone from Databricks end can look into this issue further.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-21-2024 01:56 AM
Hi @sharukh_lodhi,Thanks for reaching out! Please review the responses and let us know which best addresses your question. Your feedback is valuable to us and the community. If the response resolves your issue, kindly mark it as the accepted solution. This will help close the thread and assist others with similar queries. We appreciate your participation and are here if you need further assistance!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-02-2024 07:02 AM
I'm having a similar problem, my aim is from an Azure DataBricks notebook to invoke an AzureDataDactory pipeline I created an Access Connector for Azure Databricks to which I gave Data Factory Contributor permissions. Using these lines python
from azure.identity import ManagedIdentityCredential
credential = ManagedIdentityCredential(client_id="63d0a80e-de48-4dfe-b4ce-a1b2cb90d494")
credential.get_token("https://management.azure.com/.default")
#client_id value is yhe object id of Access Connector for Azure Databricks
I'have error
WARNING:azure.identity._internal.msal_managed_identity_client:ImdsCredential.get_token failed: ManagedIdentityCredential authentication unavailable, no response from the IMDS endpoint. invalid_request
WARNING:azure.identity._internal.decorators:ManagedIdentityCredential.get_token failed: ManagedIdentityCredential authentication unavailable, no response from the IMDS endpoint. invalid_request
any suggestion?