09-10-2021 04:29 AM
I wanted to mount a ADLG2 on databricks and take advantage on the abfss driver which should be better for large analytical workloads (is that even true in the context of DB?).
Setting an OAuth is a bit of a pain so I wanted to take the simpler approach and use a Shared Key. Following what I've seen in Hadoop documentation this should work, any ideas of what I might be doing wrong? I get a `java.lang.NullPointerException: authEndpoint` error.
%py
dbutils.fs.mount(
source = "abfss://"+ container + "@" + storageAccountName + ".dfs.core.windows.net",
mount_point = "/mnt/" + container,
extra_configs = {'fs.azure.account.auth.type.' + storageAccountName + '.dfs.core.windows.net': "SharedKey",
'fs.azure.account.key.' + storageAccountName + '.dfs.core.windows.net': storageAccountAccessKey})
10-04-2021 02:57 AM
Hi @Fernando Mendez ,
The below document will help you to mount the ADLS gen2 using abfss:
Could you please check if this helps?
09-13-2021 07:10 AM
I use OAUTH but it should be similar.
What I use (and it works):
"fs.azure.account.auth.type": "OAuth", (for you this is SharedKey I presume)
I don't think you have to pass the storage accountname in the extra_configs (or dfs.core.windows.net)
So I would try with just fs.azure.account.key and fs.azure.account.auth.type
That being said: Oauth is the way to go if you are going to a production scenario.
09-13-2021 08:25 AM
I tried changing the configs as suggested but I have the same error. I guess using Oauth would be recommended, is data governance the main benefit of Oauth?
I have managed to connect to the ADLG2 using the blob endpoint and the wasbs:// driver which will be good enough for dev.
10-04-2021 02:57 AM
Hi @Fernando Mendez ,
The below document will help you to mount the ADLS gen2 using abfss:
Could you please check if this helps?
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group