How can you mount an Azure Data Lake (gen2) using abfss and Shared Key?

[FMendez]

I wanted to mount a ADLG2 on databricks and take advantage on the abfss driver which should be better for large analytical workloads (is that even true in the context of DB?).

Setting an OAuth is a bit of a pain so I wanted to take the simpler approach and use a Shared Key. Following what I've seen in Hadoop documentation this should work, any ideas of what I might be doing wrong? I get a `java.lang.NullPointerException: authEndpoint` error.

%py
dbutils.fs.mount(
  source = "abfss://"+ container + "@" + storageAccountName + ".dfs.core.windows.net",
  mount_point = "/mnt/" + container,
  extra_configs = {'fs.azure.account.auth.type.' + storageAccountName + '.dfs.core.windows.net': "SharedKey",
                   'fs.azure.account.key.' +       storageAccountName + '.dfs.core.windows.net': storageAccountAccessKey})

[Kaniz]

Hi @ FMendez! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers on the community have an answer to your question first. Or else I will follow up shortly with a response.

[-werners-]

I use OAUTH but it should be similar.

What I use (and it works):

"fs.azure.account.auth.type": "OAuth", (for you this is SharedKey I presume)

I don't think you have to pass the storage accountname in the extra_configs (or dfs.core.windows.net)

So I would try with just fs.azure.account.key and fs.azure.account.auth.type

That being said: Oauth is the way to go if you are going to a production scenario.

[FMendez]

I tried changing the configs as suggested but I have the same error. I guess using Oauth would be recommended, is data governance the main benefit of Oauth?

I have managed to connect to the ADLG2 using the blob endpoint and the wasbs:// driver which will be good enough for dev.

[User16753724663 · Accepted solution]

Hi @Fernando Mendez​ ,

The below document will help you to mount the ADLS gen2 using abfss:

• https://docs.databricks.com/data/data-sources/azure/adls-gen2/azure-datalake-gen2-get-started.html

Could you please check if this helps?