Unable to mount ADLS gen2 to databricks file system

[Sangram]

I am unable to mount ADLS gen2 storage path into databricks storage path.

It is throwing error as unsupported azure scheme:abfss

May I know the reason.

Below are the steps that I followed: -

1. create a service principal

2. store the service principal's secret in azure key vault

3. Give service principal get and list permissions in azure key vault

4. Give service principal necessary permission "blob storage contributor" role in ADLS

Now, please tell me what additional step I need to perform to remove this error. I have followed these steps from youtube video (64) 30.Access Data Lake Storage Gen2 or Blob Storage with an Azure service principal in Azure Datab...

[Kaniz]

Hi @Sangram , Certainly! Let’s troubleshoot the issue with mounting Azure Data Lake Storage Gen2 (ADLS Gen2) into Databricks.

Azure Key Vault Permissions:

• Ensure that the Azure Databricks application has the necessary permissions on the Azure Key Vault. Specifically, you need to add an access policy in the Key Vault for the Azure Databricks application....
• To do this, navigate to your Azure Key Vault in the Azure portal, go to Access policies, and add the appropriate permissions for the Azure Databricks application.

Service Principal Configuration:

• Confirm that you’ve correctly created the service principal and stored its secret in Azure Key Vault.
• Make sure the service principal has the required permissions:
  • Blob Storage Contributor role in ADLS Gen2.
  • Application Administrator role or Application.ReadWrite.All permission in Microsoft Enterprise ID (formerly Azure Active Directory) if you create a new service principal for the storage credential.

Unity Catalog External Locations:

• Databricks recommends using Unity Catalog external locations and Azure managed identities to connect to Azure Data Lake Storage Gen2.
• An external location combines a cloud storage path with a storage credential.
• The storage credential can be either an Azure managed identity or a service principal.
• Databricks strongly recommends using an Azure managed identity for better security and management2.

Spark Properties Configuration:

• You can set Spark properties to configure Azure credentials for accessing Azure storage.
• If you’re using a service principal, ensure that the dfs.adls.oauth2.refresh.url property is correct....

Legacy Windows Azure Storage Blob Driver (WASB):

• Note that the legacy WASB driver has been deprecated, and Databricks recommends using ABFS (Azure Blob File System) instead.
• ABFS has numerous benefits over WASB. Refer to the Azure documentation on ABFS for more details2.

Migrating from Azure Data Lake Storage Gen1 to Gen2:

• Azure has announced the pending retirement of Azure Data Lake Storage Gen1.
• Databricks recommends migrating all data from Gen1 to Gen2.
• If you haven’t migrated yet, refer to the documentation on Accessing Azure Data Lake Storage Gen1 fr...2.

If you encounter any further challenges, feel free to ask for additional assistance. 😊