Trying to mount Azure Data Lake Storage Gen 2 to Azure Databricks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-20-2023 02:29 AM
I have validated many many many times all my credentials and I am still getting the following error (at the very end). ChatGPT said to basically recheck everything and I did. The one thing I didn't do was grant the permissions which I have since done and I am still having problems. Here is what I have done so far:
Ensure that the storage_account_name variable is correctly assigned with the name of your Azure Storage account. Double-check that the variable is defined and has the correct value.
Verify that the mount_point variable is defined and specifies a valid mount point path in Azure Databricks. The mount point should start with /mnt/ followed by a unique name or path.
Confirm that the configs dictionary variable is defined and populated with the necessary configuration settings. Ensure that all the required keys and values are present in the dictionary.
Check that the Azure AD application credentials (client ID, client secret, directory ID) used in the configs dictionary are valid and have the necessary permissions to access the Azure Data Lake Storage Gen2 account.
Any other ideas on what I can do? Thank you
Error:
java.lang.NullPointerException
---------------------------------------------------------------------------
ExecutionError Traceback (most recent call last)
File <command-3548438897654738>:19
10 configs = {
11 "fs.azure.account.auth.type": "OAuth",
12 "fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",
(...)
15 "fs.azure.account.oauth2.client.endpoint": "https://login.microsoftonline.com/59bfdcb9-d634-447e-88ab-e561deb22047/oauth2/token"
16 }
18 # Mount the storage account
---> 19 dbutils.fs.mount(
20 source=f"abfss://{storage_account_name}.dfs.core.windows.net/",
21 mount_point=mount_point,
22 extra_configs=configs
23 )
File /databricks/python_shell/dbruntime/dbutils.py:362, in DBUtils.FSHandler.prettify_exception_message.<locals>.f_with_exception_handling(*args, **kwargs)
360 exc.__context__ = None
361 exc.__cause__ = None
--> 362 raise exc
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-20-2023 08:19 PM
Hi @Chris Sarrico
Great to meet you, and thanks for your question!
Let's see if your peers in the community have an answer to your question. Thanks.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-20-2023 09:14 PM
Hi @Chris Sarrico
Could you please specify the container name before the storage account name like this :
so the source looks like this. :
source = "abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/"
I hope this will work