AbfsRestOperationException when adding privatelink.dfs.core.windows.net

[fabian564]

Hey Databricks forum,

Have been searching a lot, but can't find a solution. I have the following setup:

- a vnet connected to the databricks workspace with
   - public-subnet (deligated to Microsoft.Databricks/workspaces) and a NSG
   - private-subnet (deligated to Microsoft.Databricks/workspaces) and a NSG
   - private endpoints subnet

- Azure Data Lake Storage Gen2 with Hierarchical Namespace enabled
  - has a container 'metastore'
  - Public network access is disabled
  - Added private endpoint
      - Target sub-resource is dfs
      - to the private endpoints subnet
      - added dns zone privatelink.dfs.core.windows.net with a A record [name of storage resource] and also a vnet link to the vnet.

- Unity-catalog-access-connector with managed identity
 - with RBAC permissions with Storage Blob Data Contributor on the storage.

If I add a new external location with the following input:
- Storage credential - (refrenced with the ResourceId of the Unity-catalog-access-connector)
- abfss://metastore@[name of storage resource].privatelink.dfs.core.windows.net

I recieve the message:

> Failed to access cloud storage: [AbfsRestOperationException] () exceptionTraceId=ff1075e9-00d9-44b6-a602-9d7c19fbae9b

When I give the storage a public-ip and set 'staimzdatabricks.dfs.core.windows.net' it succeeds.

If I search for this exception, I get the response this could be network or permission related.
I run from the Databricks workspace a compute a nslookup for [name of storage resource].privatelink.dfs.core.windows.net:

>Server: 168.63.129.16
> Address: 168.63.129.16#53
>
> Non-authoritative answer:
> Name: [name of storage resource].privatelink.dfs.core.windows.net
> Address: 10.1.1.4

Do you have an I idea what I am missing here?

[szymon_dybczak]

Hi @fabian564 ,

Try to add also blob private endpoint. I think it's also needed for some metadata operations.

[fabian564]

Hi @szymon_dybczak ,

I added also the blob private endpoint. When you add a new Databricks external location there is only 1 URL field, which defines for `dfs` url, also tried with abfss://metastore@[name of storage resource].privatelink.blob.core.windows.net/ same result (Failed to access cloud storage: [AbfsRestOperationException]).

I see this suggestion more often, I think this is referring to the NCC configuration in Databricks, this is for the Databricks Serverless SQL Compute, there you can add private endpoint rules and is given a blob and dfs private endpoint 🙂

[Hubert-Dudek]

don't use

abfss://metastore@[name of storage resource].privatelink.dfs.core.windows.net

Just use the standard URL

abfss://metastore@<storageaccount>.dfs.core.windows.net

DNS will resolve it to a private link.

[fabian564]

Yes, that's the solution! I thought I had tested this (maybe some caching..)

When I changed it to abfss://metastore@<storageaccount>.dfs.core.windows.net it still failed with:
Failed to access cloud storage: [AbfsRestOperationException]

The storage public network access: must not be "Secured by perimeter (Most restricted)" but "Disable".
I did this before, back then I received a public-ip response with nslookup now apparently it's a private-ip:

> Server: 168.63.129.16
> Address: 168.63.129.16#53
> Non-authoritative answer:
> <storageaccount>.dfs.core.windows.net canonical name = staimzdatabricks.privatelink.dfs.core.windows.net.
> Name: staimzdatabricks.privatelink.dfs.core.windows.net
> Address: 10.1.1.4

Thank you guys! I was really out of idea's 

[szymon_dybczak]

Great news that it works now! 🙂