topic Re: Using user-assigned managed identity inside notebook in Data Engineering

Using user-assigned managed identity inside notebook

shervinmir — Tue, 18 Feb 2025 04:17:49 GMT

Hi team,

I am interested in using a user-assigned managed identity within my notebook. I've come across examples using system-assigned managed identities or leveraging the Access Connector for Azure Databricks via Unity Catalog. However, as I do not have access to my Azure subscription as a Global Administrator, Unity Catalog isn't an option for me.

I've noticed that it's possible to run the notebook as the current user or add a service principal to the workspace. I've already added the user-assigned managed identity to my workspace. However, when attempting to run the notebook using this imported managed identity, I encounter authentication errors. My goal is to connect to a Gen 2 Storage account.

I've seen similar issues discussed in the community, but I haven't found any recent solutions. I'm reaching out to inquire if there's a way to achieve my objective.

Any assistance you can provide would be greatly appreciated.

Thanks for your time,

Shervin

Re: Using user-assigned managed identity inside notebook

shervinmir — Tue, 18 Feb 2025 08:27:37 GMT

Thanks for your reply. The user-managed identity is not the issue because I use it on ADF to pull data from the same storage account. I have followed the documentation for adding the managed identity to workspace. Perhaps my issue is the syntax code used inside the notebook. Maybe I need to get a token using a library? Any example or documentation would help.

I have tried

df = spark.read.format("delta").option("header", "true").load("abfss://{base folder}@{storage account name}.dfs.core.windows.net/{path to file}")

df.show()

i have also tried to use storage SDK and also tried to mount the storage but no success

from azure.storage.filedatalake import DataLakeServiceClient

from azure.identity import ManagedIdentityCredential credential = ManagedIdentityCredential()

service_client = DataLakeServiceClient( account_url=f"https://{storage_account_name}.dfs.core.windows.net", credential=credential )

# List filesystems (containers)

for filesystem in service_client.list_file_systems(): print(filesystem.name)

Can you point me to the correct documentation? I think my issues is what is inside the notebook.

Thanks again

Re: Using user-assigned managed identity inside notebook

shervinmir — Tue, 11 Mar 2025 00:50:29 GMT

Hi team,

Just wondering if anyone has any suggestions. We are still unable to use User Assigned managed identity inside the a notebook in Databricks to connect to a external Gen 2 storage

Re: Using user-assigned managed identity inside notebook

damour-jrs — Fri, 14 Mar 2025 16:30:16 GMT

I have followed the documentation for adding the managed identity to workspace.

you didn't link the guides you followed. but if it was https://learn.microsoft.com/en-us/azure/databricks/dev-tools/auth/azure-mi or https://learn.microsoft.com/en-us/azure/databricks/dev-tools/azure-mi-auth those are all about granting a UMI the ability to access a workspace...not about letting a workspace access other azure resources so they won't help AFAIK

Re: Using user-assigned managed identity inside notebook

damour-jrs — Fri, 14 Mar 2025 17:02:11 GMT

i also came across https://community.databricks.com/t5/data-engineering/how-to-assign-user-managed-identity-to-dbr-cluster-so-i-can-use/td-p/83909 which says...u have to use unity catalog's storage credential abstraction to make use of a umi...that makes a lot of sense as it abstracts datarbicks from the specific provider..eg UMI means nothing in AWS.