topic Can we use Managed Identity to create mount point for ADLS Gen2 in Data Engineering

Can we use Managed Identity to create mount point for ADLS Gen2

srikanth2 — Fri, 03 May 2024 19:35:45 GMT

Hi,

We would like to use Azure Managed Identity to create mount point to read/write data from/to ADLS Gen2?

We are also using following code snippet to use MSI authentication to read data from ADLS Gen2 but it is giving error,

storage_account_name = "<<mystorageaccount>>"

storage_endpoint = "abfss://{}/".format(storage_account_name)

spark.conf.set("fs.azure.account.auth.type.{}.dfs.core.windows.net".format(storage_account_name), "MSI")

spark.conf.set("fs.azure.account.oauth.provider.type.{}.dfs.core.windows.net".format(storage_account_name), "org.apache.hadoop.fs.azurebfs.oauth2.MSICredentialProvider")

Error:

"No enum constant shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.services.AuthType.MSI"

It looks like MSI support is not enabled. Can we use Managed Identity(User Assigned Managed Identity) to read/write from ADLS Gen2 inside notebook, not through unified catalog.

Thanks

Re: Can we use Managed Identity to create mount point for ADLS Gen2

Walter_C — Sat, 04 May 2024 16:23:33 GMT

It seems that using User Assigned Managed Identity to read/write from ADLS Gen2 inside a notebook is not directly supported at the moment.

Re: Can we use Managed Identity to create mount point for ADLS Gen2

srikanth2 — Sat, 04 May 2024 20:25:22 GMT

Thanks for the Reply Walter.

Can I use Subject Name + Issuer Authentication inside the Databricks notebook? Is this supported?

Can I use Federated Identity Credential inside the notebook?

If yes, can you please refer me right pointers.

Thanks