topic Trouble on Accessing Azure Storage from Databricks (Python) in Data Engineering

Trouble on Accessing Azure Storage from Databricks (Python)

Geoff123 — Tue, 26 Mar 2024 19:52:01 GMT

I used the same accessing method shown in https://community.databricks.com/t5/data-engineering/to-read-data-from-azure-storage/td-p/32230 but kept get the error below.

org.apache.spark.SparkSecurityException: [INSUFFICIENT_PERMISSIONS] Insufficient privileges:

Any help would be greatly appreciated.

Re: Trouble on Accessing Azure Storage from Databricks (Python)

Wojciech_BUK — Wed, 27 Mar 2024 08:10:57 GMT

Hello,
I will try to help you, but need some more details:

1) Are you using Unity Catalog on your workspace?

2) What is your storage account Firewall configuration setup?

3) Which exactly method you are using ? (Do you mount it? Or do you set spark config)?

4) Which authentication method you are using? SAS / KEY / OAUTH ?

Re: Trouble on Accessing Azure Storage from Databricks (Python)

Wojciech_BUK — Wed, 27 Mar 2024 08:27:55 GMT

Hi Geoff123,

you can also take a look on your Storage Account firewall and make sure it is accessiable by Databricks Cluster.

In addition to this, if you are using Unity Catalog, the access pattern to storage will be different.

Wojciech

Re: Trouble on Accessing Azure Storage from Databricks (Python)

Geoff123 — Thu, 28 Mar 2024 13:42:10 GMT

Wojciech,

Thank you for your reply! Here are the answers of your questions.

1) Are you using Unity Catalog on your workspace? Yes

2) What is your storage account Firewall configuration setup?

Where can I find the Firewall configuration in my storage account?

3) Which exactly method you are using ? (Do you mount it? Or do you set spark config)?

I tried both mount and set spark config but got the same error.

4) Which authentication method you are using? SAS / KEY / OAUTH ?

I used the access key of the storage account to authenticate.

Thanks again!

Re: Trouble on Accessing Azure Storage from Databricks (Python)

Geoff123 — Thu, 28 Mar 2024 13:45:06 GMT

Hi Kaniz,

Thank you for your help!

Yes, I used the correct syntax for the file location.

source = "wasbs://"+blob_container+"@"+storage_account_name+".blob.core.windows.net/sub_folder_name/file_name.csv"

Re: Trouble on Accessing Azure Storage from Databricks (Python)

Geoff123 — Thu, 28 Mar 2024 13:51:33 GMT

Wojciech,

Do you have any helpful tutorial to show how to check that Databricks Cluster is accessible to the Storage Account Firewall?

Thanks!

Re: Trouble on Accessing Azure Storage from Databricks (Python)

Wojciech_BUK — Thu, 28 Mar 2024 13:56:47 GMT

Hi,

you can find storage account firewall information by accessing resource in azure portal

Please mind that if you are using Unity Catalog you should NOT mount Storage Account, you should rather use abstraction of Storage Creadentials and External Location (optionally External Volumes)

Re: Trouble on Accessing Azure Storage from Databricks (Python)

Geoff123 — Thu, 28 Mar 2024 20:41:39 GMT

I have the same settings as yours in Networking settings.