VNet Injected Workspace trouble connecting to the Storage Account of a catalog (UC),

[dbdev]

Hi! 

I have some issues with setting up my workspace and storage account within a virtual network and letting them connect to each other.

Background set up (all done in terraform):

- Databricks workspace with VNet Injection and Unity Catalog enabled. (3 subnets: 2 delegated for databricks workspace, 1 subnet for private endpoint to do front-end private link)

- One azure storage account (public access disabled) with a private endpoint that uses a new subnet within the same vnet, (subresource type 'blob')

- Databricks Access Connector with Storage Blob Data Contributor role assigned to the storage account

- An external location using the access connector as storage credential, pointing to the storage account and used as root for my catalog 

Issue: 

I get the following errors when starting up my dlt (not serverless) or running a query on a sql warehouse that writes data to my catalog: 

- Failed to initialize the UCContext

- DLT ERROR CODE: EXECUTION_SERVICE_STARTUP_FAILURE.AZURE_STORAGE_PERMISSION_ISSUE

- 403, This Azure storage request is not authorized. The storage account's 'Firewalls and virtual networks' settings may be blocking access to storage services. Please verify your Azure storage credentials or firewall exception settings.

It seems there is still an issue with my compute not being able to connect to my storage account. 
Does anyone know how to resolve this? What am I missing?
It seems like my clusters are not using the private endpoint to connect to the storage account.

Thank you in advance.

[szymon_dybczak]

Hi @dbdev ,

First, could you check if you didn't forget to link private DNS zone to your VNET? Second, check if you have proper entries in your private DNS zone. Now, it looks like your databricks compute is trying to use public endpoint instead of private one

[dbdev]

That's the only part I didn't do myself, it was automatically added by a policy. When I go to DNS configuration of my storage endpoint I see this:

So it seems a private DNS zone is added, and there is a record. Is this sufficient?

[szymon_dybczak]

@dbdev  It looks correct, so that means your azure policy worked correctly. But you need to also check if that Private DNS Zone is linked to your VNET. Go to Private DNS Zone -> DNS Management -> Virtual Network Links. Your VNET should be linked there (if not that could be a problem)

[dbdev · Accepted solution]

It was correctly linked.

Turned out we were missing one extra private endpoint of type 'dfs'.
So for our storage account we needed to create 2 private endpoints, both configured on the same subnet with one of subresource type 'blob' and one subresource type 'dfs'.

We still have an issue with connecting to it via serverless, but that probably needs some NCC setup in the account console.

Thanks for the help!