topic Re: How can you mount an Azure Data Lake (gen2) using abfss and Shared Key? in Data Engineering

How can you mount an Azure Data Lake (gen2) using abfss and Shared Key?

FMendez — Fri, 10 Sep 2021 11:29:06 GMT

I wanted to mount a ADLG2 on databricks and take advantage on the abfss driver which should be better for large analytical workloads (is that even true in the context of DB?).

Setting an OAuth is a bit of a pain so I wanted to take the simpler approach and use a Shared Key. Following what I've seen in Hadoop documentation this should work, any ideas of what I might be doing wrong? I get a `java.lang.NullPointerException: authEndpoint` error.

%py
dbutils.fs.mount(
  source = "abfss://"+ container + "@" + storageAccountName + ".dfs.core.windows.net",
  mount_point = "/mnt/" + container,
  extra_configs = {'fs.azure.account.auth.type.' + storageAccountName + '.dfs.core.windows.net': "SharedKey",
                   'fs.azure.account.key.' +       storageAccountName + '.dfs.core.windows.net': storageAccountAccessKey})

Re: How can you mount an Azure Data Lake (gen2) using abfss and Shared Key?

-werners- — Mon, 13 Sep 2021 14:10:24 GMT

I use OAUTH but it should be similar.

What I use (and it works):

"fs.azure.account.auth.type": "OAuth", (for you this is SharedKey I presume)

I don't think you have to pass the storage accountname in the extra_configs (or dfs.core.windows.net)

So I would try with just fs.azure.account.key and fs.azure.account.auth.type

That being said: Oauth is the way to go if you are going to a production scenario.

Re: How can you mount an Azure Data Lake (gen2) using abfss and Shared Key?

FMendez — Mon, 13 Sep 2021 15:25:21 GMT

I tried changing the configs as suggested but I have the same error. I guess using Oauth would be recommended, is data governance the main benefit of Oauth?

I have managed to connect to the ADLG2 using the blob endpoint and the wasbs:// driver which will be good enough for dev.

Re: How can you mount an Azure Data Lake (gen2) using abfss and Shared Key?

User16753724663 — Mon, 04 Oct 2021 09:57:28 GMT

Hi @Fernando Mendez ,

The below document will help you to mount the ADLS gen2 using abfss:

https://docs.databricks.com/data/data-sources/azure/adls-gen2/azure-datalake-gen2-get-started.html

Could you please check if this helps?