cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Use of private endpoints for storage in workspace with EnableNoPublicIP=Yes and VnetInjection=No

Greg_Galloway
New Contributor III
  • We know that Databricks with VNET injection (our own VNET) allows is to connect to ADLS Gen2 over private endpoints. This is what we typically do.
  • We have a customer who created Databricks with EnableNoPublicIP=Yes (secure cluster connectivity) and VnetInjection=No. So itโ€™s using a managed VNET in the Databricks managed resource group. Weโ€™re wondering if we can make it connect to ADLS Gen2 over private endpoints. We havenโ€™t been successful but are close. Do we need to delete and recreate the Databricks workspace with VNET injection?
  • Weโ€™ve created a VNET peering in Databricks to MyVNET and a VNET peering from MyVNET the other end back to the Databricks managed VNET
  • Private endpoint is created for ADLS Gen2 in MyVNET and private DNS zone is setup with a VNET link to MyVNET. A VM in MyVNET can resolve DNS to the private endpoint 10.0.0.5 private IP and connect fine.
  • In a Databricks cluster I can successfully connect to 10.0.0.5 (the private endpoint IP) and have validated this with %sh nc -zv 10.0.0.5 443 which connects successfully
  • However private DNS resolution isn't working. If I run %sh nslookup mystorageaccount.dfs.core.windows.net it returns the public IP address
  • The reason the DNS resolution isnโ€™t happening is because in my Private DNS Zone I am unable to add a VNET link to the Databricks managed VNET. When I try to do that I get the typical error that I can't make changes to anything in the managed resource group databricks-rg-XXXXXX
  • So basically my question is whether thereโ€™s any way to add a Private DNS Zone virtual network link to a Databricks managed VNET (no public IP=yes) other than configuring custom DNS? https://docs.microsoft.com/en-us/azure/databricks/kb/cloud/custom-dns-routing

1 ACCEPTED SOLUTION

Accepted Solutions

User16871418122
Contributor III

Managed VNET is locked and allows very limited config tuning like VNET peering that too facilitated and needs to be done from Databricks UI. If they want more control on VNET they need to migrate to VNET injected workspace.

View solution in original post

5 REPLIES 5

Kaniz
Community Manager
Community Manager

Hi Greg Galloway! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will get back to you soon. Thanks.

Hubert-Dudek
Esteemed Contributor III

I use mainly exact the same setup. Please go to virtual network and check subnets. In PrivateDatabricks check is there "Service endpoints

Create service endpoint policies to allow traffic to specific azure resources from your virtual network over service endpoints" and there you should see it. You should also see ServiceEndpoints subnet.

image

Greg_Galloway
New Contributor III

@Hubert Dudekโ€‹ Thanks for taking the time to reply. I think we're talking apples and oranges unfortunately. I didn't do VNET injection so the VNET and subnets in question are a *managed* VNET in the Databricks managed resource group. You are not allowed to make any changes to it directly. What you describe is bringing your own VNET which works perfectly, of course. Any idea how to make this work when VnetInjection=No?

User16871418122
Contributor III

Managed VNET is locked and allows very limited config tuning like VNET peering that too facilitated and needs to be done from Databricks UI. If they want more control on VNET they need to migrate to VNET injected workspace.

Thanks @Gobinath Viswanathanโ€‹ . We'll move to VNET injection for now. Hopefully at some point the Databricks UI will provide additional control to make this scenario workable without bringing our own VNET.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.