Connecting to DataLake Gen2 from Azure Databricks with Private Endpoint

alpha
New Contributor III

Hi,

I have datalake gen2 with vnet and private endpoint. I do have databricks workspace in same vnet. I am trying to access the datalake from databricks but I keep getting error when I allow access only for selected network in datalake. I get error when I try to mount the datalake. It works when I allow access for all network. I do have one subnet with service endpoint as Micrsoft.storage. Do I need to do this?

I do not have NGS setup for subnet. Is firewall blocking this? Can you please me fix this issue. I read various blogs but its not helping.

When I tried doing nslookup for strorage blob I can see its using subnet and private endpoint to connect but when I try same thing for datalake, it does not look like private endpoint is being used for connect. (I see private IP as 20.x.x.x but my vnet start from 10.x.x.x). I did setup private end point for blob and dfs separately.

Here is error message.

Operation failed: "This request is not authorized to perform this operation.", 403, PUT, https://xxxxxxxxxxx.dfs.core.windows.net/test?resource=filesystem, AuthorizationFailure, "This request is not authorized to perform this operation. RequestId:3f54eba1-601f-0075-75acccccc8fcc00000 Time:2022-06-21T18:57:54.4141801Z"

Hubert-Dudek
Databricks MVP

You can always check the tutorial regarding ADLS and private link under this link https://community.databricks.com/s/feed/0D53f00001eQGOHCA4


My blog: https://databrickster.medium.com/