Use of private endpoints for storage in workspace with EnableNoPublicIP=Yes and VnetInjection=No

[Greg_Galloway]

• We know that Databricks with VNET injection (our own VNET) allows is to connect to ADLS Gen2 over private endpoints. This is what we typically do.
• We have a customer who created Databricks with EnableNoPublicIP=Yes (secure cluster connectivity) and VnetInjection=No. So it’s using a managed VNET in the Databricks managed resource group. We’re wondering if we can make it connect to ADLS Gen2 over private endpoints. We haven’t been successful but are close. Do we need to delete and recreate the Databricks workspace with VNET injection?
• We’ve created a VNET peering in Databricks to MyVNET and a VNET peering from MyVNET the other end back to the Databricks managed VNET
• Private endpoint is created for ADLS Gen2 in MyVNET and private DNS zone is setup with a VNET link to MyVNET. A VM in MyVNET can resolve DNS to the private endpoint 10.0.0.5 private IP and connect fine.
• In a Databricks cluster I can successfully connect to 10.0.0.5 (the private endpoint IP) and have validated this with %sh nc -zv 10.0.0.5 443 which connects successfully
• However private DNS resolution isn't working. If I run %sh nslookup mystorageaccount.dfs.core.windows.net it returns the public IP address
• The reason the DNS resolution isn’t happening is because in my Private DNS Zone I am unable to add a VNET link to the Databricks managed VNET. When I try to do that I get the typical error that I can't make changes to anything in the managed resource group databricks-rg-XXXXXX
• So basically my question is whether there’s any way to add a Private DNS Zone virtual network link to a Databricks managed VNET (no public IP=yes) other than configuring custom DNS? https://docs.microsoft.com/en-us/azure/databricks/kb/cloud/custom-dns-routing

[Hubert-Dudek]

I use mainly exact the same setup. Please go to virtual network and check subnets. In PrivateDatabricks check is there "Service endpoints

Create service endpoint policies to allow traffic to specific azure resources from your virtual network over service endpoints" and there you should see it. You should also see ServiceEndpoints subnet.

[Greg_Galloway]

@Hubert Dudek​ Thanks for taking the time to reply. I think we're talking apples and oranges unfortunately. I didn't do VNET injection so the VNET and subnets in question are a *managed* VNET in the Databricks managed resource group. You are not allowed to make any changes to it directly. What you describe is bringing your own VNET which works perfectly, of course. Any idea how to make this work when VnetInjection=No?

[User16871418122 · Accepted solution]

Managed VNET is locked and allows very limited config tuning like VNET peering that too facilitated and needs to be done from Databricks UI. If they want more control on VNET they need to migrate to VNET injected workspace.

[Greg_Galloway]

Thanks @Gobinath Viswanathan​ . We'll move to VNET injection for now. Hopefully at some point the Databricks UI will provide additional control to make this scenario workable without bringing our own VNET.