Trouble on Accessing Azure Storage from Databricks (Python)

[Geoff123]

I used the same accessing method shown in https://community.databricks.com/t5/data-engineering/to-read-data-from-azure-storage/td-p/32230 but kept get the error below.

org.apache.spark.SparkSecurityException: [INSUFFICIENT_PERMISSIONS] Insufficient privileges:

Any help would be greatly appreciated.

[Wojciech_BUK]

Hello,
I will try to help you, but need some more details:

1) Are you using Unity Catalog on your workspace?

2) What is your storage account Firewall configuration setup?

3) Which exactly method you are using ? (Do you mount it? Or do you set spark config)?

4) Which authentication method you are using? SAS / KEY / OAUTH ?

[Geoff123]

Wojciech,

Thank you for your reply!  Here are the answers of your questions.

1) Are you using Unity Catalog on your workspace? Yes

2) What is your storage account Firewall configuration setup?

Where can I find the Firewall configuration in my storage account?

3) Which exactly method you are using ? (Do you mount it? Or do you set spark config)?

I tried both mount and set spark config but got the same error.

4) Which authentication method you are using? SAS / KEY / OAUTH ?

I used the access key of the storage account to authenticate.

Thanks again!

[Kaniz]

Hi @Geoff123, 

• When reading data from Azure Blob Storage in Databricks, ensure that you use the correct protocol for the file path reference. Instead of using the local file system path, use the “wasbs” protocol.
• The correct format for accessing files in Azure Blob Storage is: wasbs://<container-name>/<path-to-file>.
• Replace <container-name> with the actual name of your Blob storage container and <path-to-file> with the relative path to your Excel file within the container.

[Geoff123]

Hi Kaniz,

Thank you for your help!

Yes, I used the correct syntax for the file location.

source = "wasbs://"+blob_container+"@"+storage_account_name+".blob.core.windows.net/sub_folder_name/file_name.csv"

[Wojciech_BUK]

Hi Geoff123,

you can also take a look on your Storage Account firewall and make sure it is accessiable by Databricks Cluster.

In addition to this, if you are using Unity Catalog, the access pattern to storage will be different.

Wojciech

[Geoff123]

Wojciech,

Do you have any helpful tutorial to show how to check that Databricks Cluster is accessible to the Storage Account Firewall?

Thanks!

[Wojciech_BUK]

Hi,

you can find storage account firewall information by accessing resource in azure portal

Please mind that if you are using Unity Catalog you should NOT mount Storage Account, you should rather use abstraction of Storage Creadentials and External Location (optionally External Volumes)  

[Geoff123]

I have the same settings as yours in Networking settings.