cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Trying to mount Azure Data Lake Storage Gen 2 to Azure Databricks

ChrisS
New Contributor III

I have validated many many many times all my credentials and I am still getting the following error (at the very end). ChatGPT said to basically recheck everything and I did. The one thing I didn't do was grant the permissions which I have since done and I am still having problems. Here is what I have done so far:

Ensure that the storage_account_name variable is correctly assigned with the name of your Azure Storage account. Double-check that the variable is defined and has the correct value.

Verify that the mount_point variable is defined and specifies a valid mount point path in Azure Databricks. The mount point should start with /mnt/ followed by a unique name or path.

Confirm that the configs dictionary variable is defined and populated with the necessary configuration settings. Ensure that all the required keys and values are present in the dictionary.

Check that the Azure AD application credentials (client ID, client secret, directory ID) used in the configs dictionary are valid and have the necessary permissions to access the Azure Data Lake Storage Gen2 account.

Any other ideas on what I can do? Thank you

Error:

java.lang.NullPointerException

---------------------------------------------------------------------------

ExecutionError Traceback (most recent call last)

File <command-3548438897654738>:19

10 configs = {

11 "fs.azure.account.auth.type": "OAuth",

12 "fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",

(...)

15 "fs.azure.account.oauth2.client.endpoint": "https://login.microsoftonline.com/59bfdcb9-d634-447e-88ab-e561deb22047/oauth2/token"

16 }

18 # Mount the storage account

---> 19 dbutils.fs.mount(

20 source=f"abfss://{storage_account_name}.dfs.core.windows.net/",

21 mount_point=mount_point,

22 extra_configs=configs

23 )

File /databricks/python_shell/dbruntime/dbutils.py:362, in DBUtils.FSHandler.prettify_exception_message.<locals>.f_with_exception_handling(*args, **kwargs)

360 exc.__context__ = None

361 exc.__cause__ = None

--> 362 raise exc

2 REPLIES 2

Anonymous
Not applicable

Hi @Chris Sarricoโ€‹ 

Great to meet you, and thanks for your question!

Let's see if your peers in the community have an answer to your question. Thanks.

User16752245772
Contributor

Hi @Chris Sarricoโ€‹ 

Could you please specify the container name before the storage account name like this :

so the source looks like this. :

source = "abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/"

https://learn.microsoft.com/en-us/azure/databricks/dbfs/mounts#--mount-adls-gen2-or-blob-storage-wit...

I hope this will work