Databricks Community

cyborg-de · ‎09-24-2024

Hi,

I have created a new databricks workspace in Azure with backend private link.

Settings are

Required NSG rules - No Azure Databricks Rule

NSG rules for AAD and azfrontdoor were added as per documentation.

Private endpoint with subresource databricks_ui_api inside private dns zone "privatelink.azuredatabricks.net" was created successfully and I can see DNS record created correctly.

However, when I try to create a compute cluster it fails with below error.

I think it is happening because our DNS zone is created in a separate VNET and Subscription. There is peering between dns zone vnet and workspace vnet but not sure what else needs to be done to make dns resolution work.

Message
Failed to add 2 workers to the compute. Will attempt retry: false. Reason: Control Plane Request Failure

Help
Failed to get instance bootstrap steps from the Databricks Control Plane. Please check that instances have connectivity to the Databricks Control Plane.
Instance bootstrap failed command: GetRunbook

Followed by DNS resolution which does not look right.

mark_ott · 4 weeks ago

Based on your description, the error when creating a Databricks compute cluster in Azure with Private Link is likely due to DNS resolution issues between the workspace VNET and the separate VNET hosting your private DNS zone. Even with VNET peering and the correct DNS record in the private DNS zone, there are a few additional steps to ensure compute nodes can resolve "privatelink.azuredatabricks.net" via your custom DNS setup.

Key Problem

Databricks compute VMs are likely using Azure's default DNS servers, not your custom private DNS zone, so they cannot resolve the private endpoint. This is common when:

The private DNS zone is linked only to its own VNET, not directly to the workspace VNET.
Peered VNETs do not automatically make DNS zones available, and DNS queries don't transit peering by default.
Databricks documentation specifies that compute nodes must be able to resolve the private DNS zone even if it’s in another VNET.

Solutions and Next Steps

1. DNS Forwarding/Custom DNS Server

Deploy a custom DNS forwarder (e.g., Azure DNS Forwarder in your DNS VNET).
Configure your workspace VNET’s DHCP (or explicitly in VNET settings) to specify this custom DNS server. This lets the Databricks compute nodes use that custom DNS resolver and look up the private endpoint.
Make sure required DNS ports (UDP/TCP 53) are permitted in the NSGs.

2. Private DNS Zone Linking

Link the private DNS zone not just to one VNET, but also to the Databricks workspace VNET directly, even if peered.
- In the Azure Portal > Private DNS Zone > Virtual Network Links > Add, select the workspace VNET.
- This ensures records in "privatelink.azuredatabricks.net" resolve in both VNETs.
Azure VNET peering does not share private DNS zones by default.

3. Check Name Resolution on Compute VMs

Deploy a test VM in the workspace VNET.
Use nslookup to test resolution of your Databricks private endpoint FQDN. It should resolve to the private IP, not the public one.
If it fails, the above DNS forwarding or zone linking steps weren’t fully successful.

4. Databricks Control Plane Public IP Allowlisting

Make sure your NSG allows outbound HTTPS/443 to the required Databricks Control Plane public IPs (or ranges), which may be required for workspace registration and operations.

Azure Official Guidance

Summary Table – Quick Actions

Step	Required?	Notes
Direct DNS zone link	Yes	Link DNS Zone to workspace VNET
DNS forwarder	Optional	Needed if you require custom DNS logic/rules
NSG for DNS traffic	Yes	Allow DNS (53) between workspace <-> DNS VNET
Control plane outbound	Yes	Allow HTTPS to Databricks Control Plane IPs

Check and apply the steps above. Linking the private DNS zone to the workspace VNET and confirming DNS flows are the most commonly missed requirements in this scenario.