Connecting to DataLake Gen2 from Azure Databricks with Private Endpoint

[alpha]

Hi,

I have datalake gen2 with vnet and private endpoint. I do have databricks workspace in same vnet. I am trying to access the datalake from databricks but I keep getting error when I allow access only for selected network in datalake. I get error when I try to mount the datalake. It works when I allow access for all network. I do have one subnet with service endpoint as Micrsoft.storage. Do I need to do this?

I do not have NGS setup for subnet. Is firewall blocking this? Can you please me fix this issue. I read various blogs but its not helping.

When I tried doing nslookup for strorage blob I can see its using subnet and private endpoint to connect but when I try same thing for datalake, it does not look like private endpoint is being used for connect. (I see private IP as 20.x.x.x but my vnet start from 10.x.x.x). I did setup private end point for blob and dfs separately.

Here is error message.

Operation failed: "This request is not authorized to perform this operation.", 403, PUT, https://xxxxxxxxxxx.dfs.core.windows.net/test?resource=filesystem, AuthorizationFailure, "This request is not authorized to perform this operation. RequestId:3f54eba1-601f-0075-75acccccc8fcc00000 Time:2022-06-21T18:57:54.4141801Z"

[alpha · Accepted solution]

This linked helped resolved issue

https://stackoverflow.com/questions/68943377/is-it-possible-to-connect-to-azure-storage-account-with...

[Kaniz]

Hi @alpha222 alpha​, Thank you for getting back here with a solution. Would you like to mark your answer as the best as it would greatly help our community?

[Hubert-Dudek]

You can always check the tutorial regarding ADLS and private link under this link https://community.databricks.com/s/feed/0D53f00001eQGOHCA4