Trying to mount Azure Data Lake Storage Gen 2 to Azure Databricks

ChrisS — Tue, 20 Jun 2023 09:29:46 GMT

I have validated many many many times all my credentials and I am still getting the following error (at the very end). ChatGPT said to basically recheck everything and I did. The one thing I didn't do was grant the permissions which I have since done and I am still having problems. Here is what I have done so far:

Ensure that the storage_account_name variable is correctly assigned with the name of your Azure Storage account. Double-check that the variable is defined and has the correct value.

Verify that the mount_point variable is defined and specifies a valid mount point path in Azure Databricks. The mount point should start with /mnt/ followed by a unique name or path.

Confirm that the configs dictionary variable is defined and populated with the necessary configuration settings. Ensure that all the required keys and values are present in the dictionary.

Check that the Azure AD application credentials (client ID, client secret, directory ID) used in the configs dictionary are valid and have the necessary permissions to access the Azure Data Lake Storage Gen2 account.

Any other ideas on what I can do? Thank you

Error:

java.lang.NullPointerException

---------------------------------------------------------------------------

ExecutionError Traceback (most recent call last)

File <command-3548438897654738>:19

10 configs = {

11 "fs.azure.account.auth.type": "OAuth",

12 "fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",

(...)

15 "fs.azure.account.oauth2.client.endpoint": "https://login.microsoftonline.com/59bfdcb9-d634-447e-88ab-e561deb22047/oauth2/token"

16 }

18 # Mount the storage account

---> 19 dbutils.fs.mount(

20 source=f"abfss://{storage_account_name}.dfs.core.windows.net/",

21 mount_point=mount_point,

22 extra_configs=configs

23 )

File /databricks/python_shell/dbruntime/dbutils.py:362, in DBUtils.FSHandler.prettify_exception_message.<locals>.f_with_exception_handling(*args, **kwargs)

360 exc.__context__ = None

361 exc.__cause__ = None

--> 362 raise exc

Re: Trying to mount Azure Data Lake Storage Gen 2 to Azure Databricks

Anonymous — Wed, 21 Jun 2023 03:19:28 GMT

Hi @Chris Sarrico

Great to meet you, and thanks for your question!

Let's see if your peers in the community have an answer to your question. Thanks.

Re: Trying to mount Azure Data Lake Storage Gen 2 to Azure Databricks

User16752245772 — Wed, 21 Jun 2023 04:14:51 GMT

Hi @Chris Sarrico

Could you please specify the container name before the storage account name like this :

so the source looks like this. :

source = "abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/"

https://learn.microsoft.com/en-us/azure/databricks/dbfs/mounts#--mount-adls-gen2-or-blob-storage-with-abfs

I hope this will work

topic Trying to mount Azure Data Lake Storage Gen 2 to Azure Databricks in Data Engineering

Trying to mount Azure Data Lake Storage Gen 2 to Azure Databricks

Re: Trying to mount Azure Data Lake Storage Gen 2 to Azure Databricks

Re: Trying to mount Azure Data Lake Storage Gen 2 to Azure Databricks