Use of private endpoints for storage in workspace with EnableNoPublicIP=Yes and VnetInjection=No

Greg_Galloway
New Contributor III

‎10-18-2021 11:57 AM

  • We know that Databricks with VNET injection (our own VNET) allows is to connect to ADLS Gen2 over private endpoints. This is what we typically do.
  • We have a customer who created Databricks with EnableNoPublicIP=Yes (secure cluster connectivity) and VnetInjection=No. So it’s using a managed VNET in the Databricks managed resource group. We’re wondering if we can make it connect to ADLS Gen2 over private endpoints. We haven’t been successful but are close. Do we need to delete and recreate the Databricks workspace with VNET injection?
  • We’ve created a VNET peering in Databricks to MyVNET and a VNET peering from MyVNET the other end back to the Databricks managed VNET
  • Private endpoint is created for ADLS Gen2 in MyVNET and private DNS zone is setup with a VNET link to MyVNET. A VM in MyVNET can resolve DNS to the private endpoint 10.0.0.5 private IP and connect fine.
  • In a Databricks cluster I can successfully connect to 10.0.0.5 (the private endpoint IP) and have validated this with %sh nc -zv 10.0.0.5 443 which connects successfully
  • However private DNS resolution isn't working. If I run %sh nslookup mystorageaccount.dfs.core.windows.net it returns the public IP address
  • The reason the DNS resolution isn’t happening is because in my Private DNS Zone I am unable to add a VNET link to the Databricks managed VNET. When I try to do that I get the typical error that I can't make changes to anything in the managed resource group databricks-rg-XXXXXX
  • So basically my question is whether there’s any way to add a Private DNS Zone virtual network link to a Databricks managed VNET (no public IP=yes) other than configuring custom DNS? https://docs.microsoft.com/en-us/azure/databricks/kb/cloud/custom-dns-routing

Labels:
0 Kudos