topic Re: Databricks App in Azure Databricks with private link cluster (no Public IP) in Administration & Architecture

Databricks App in Azure Databricks with private link cluster (no Public IP)

Behwar — Fri, 17 Jan 2025 12:01:56 GMT

Hello,
I've deployed Azure Databricks with a standard Private Link setup (no public IP). Everything works as expected—I can log in via the private/internal network, create clusters, and manage workloads without any issues.

When I create a Databricks App, it generates a URL like: <name>.azure.databricksapps.com

Since I didn’t initially have a Private DNS Zone for azure.databricksapps.com, my system resolved this address to a public IP. To fix this, I:

Created a Private DNS Zone for azure.databricksapps.com.
Added an A record pointing <name>.azure.databricksapps.com to my Databricks workspace private IP endpoint (same as used in privatelink.azuredatabricks.net for this workspace).

Behavior Before Adding the Private DNS Zone:
nslookup <app-name>.azure.databricksapps.com → Resolved to a public IP.
curl or accessing via a browser resulted in:
{"X-Databricks-Reason-Phrase":"Public access is not allowed for workspace: xyz"}

Behavior After Adding the Private DNS Zone:
nslookup <app-name>.azure.databricksapps.com → Now resolves to the private IP (as expected).
However, curl and browser requests still go through the public IP and return the same error:
{"X-Databricks-Reason-Phrase":"Public access is not allowed for workspace: xyz"}

Is additional configuration needed to ensure Databricks Apps work over Private Link?
Does this feature require a Public IP, or should it work fully within a private network?

Re: Databricks App in Azure Databricks with private link cluster (no Public IP)

Alberto_Umana — Fri, 17 Jan 2025 12:40:38 GMT

Hello @Behwar,

Did you make sure that your internal DNS is configured to map the web application workspace URL to your front-end VPC endpoint. This involves creating an A-record in your internal DNS that maps the workspace URL directly to the front-end (workspace) VPC endpoint private IP

Re: Databricks App in Azure Databricks with private link cluster (no Public IP)

Behwar — Fri, 17 Jan 2025 13:12:56 GMT

I'm using Azure, so instead of a VPC endpoint, I'm working with Azure Private Link. Here's what I checked and did:

Verified my Databricks workspace private endpoint under privatelink.azuredatabricks.net.
Created a Private DNS Zone for azure.databricksapps.com and mapped <app-name> to the same private IP as my Databricks workspace.
Linked my VNet to the Private DNS Zone so all internal resources resolve it correctly.
Confirmed that nslookup now returns the private IP, but browser and curl still attempt to route via the public IP.

Re: Databricks App in Azure Databricks with private link cluster (no Public IP)

sparkplug — Thu, 20 Feb 2025 08:38:40 GMT

I am also facing the same issue. Is the recommendation to create one A-record for each app, which sounds a bit cumbersome.

Re: Databricks App in Azure Databricks with private link cluster (no Public IP)

MariuszK — Fri, 21 Feb 2025 19:10:18 GMT

Do you have a private endpoint for databricks_ui_api ? You need to establish a private endpoint for users to access web app.

Re: Databricks App in Azure Databricks with private link cluster (no Public IP)

rugger-bricks — Mon, 14 Apr 2025 17:24:47 GMT

Behwar : you should have to create a specific private DNS zone for azure.databricksapps.com - if you do a nslookup on your apps url - you will see that it points to your workspace. In Azure using Azure (recursive) DNS you can see an important behavior that will give you a clue on how to handle apps with a workspace that has no public access that uses a private endpoint for access.
If I do a nslookup -debug on my apps URL, the url points to a canonical (CNAME) of the databricks workspace URL, that URL points to a canonical (CNAME) to the privatelink URL which depending on if I have a private DNS zone or forwarding rule should either stop there and resolve to a private IP address or if not continue on to point to the regional control plane endpoint.

Example : I have my app "yoworlld" and if I do a nslookup -debug without a private DNS zone you can see this behavior :

% nslookup -debug yoworld-7338663619064447.7.azure.databricksapps.com Server: 10.248.117.1 Address: 10.248.117.1#53 ------------ QUESTIONS: yoworld-7338663619064447.7.azure.databricksapps.com, type = A, class = IN ANSWERS: -> yoworld-7338663619064447.7.azure.databricksapps.com canonical name = adb-7338663619064447.7.azuredatabricks.net. ttl = 300 -> adb-7338663619064447.7.azuredatabricks.net canonical name = adb-7338663619064447.7.privatelink.azuredatabricks.net. ttl = 300 -> adb-7338663619064447.7.privatelink.azuredatabricks.net canonical name = westus.azuredatabricks.net. ttl = 300 -> westus.azuredatabricks.net internet address = 13.91.180.32 ttl = 41

Same URL with private endpoints and a private DNS zone for privatelink.azuredatabricks.net linked to my vnet :

> yoworld-7338663619064447.7.azure.databricksapps.com Server:168.63.129.16 Address:168.63.129.16#53 Non-authoritative answer: yoworld-7338663619064447.7.azure.databricksapps.com canonical name = adb-7338663619064447.7.azuredatabricks.net. adb-7338663619064447.7.azuredatabricks.net canonical name = adb-7338663619064447.7.privatelink.azuredatabricks.net. Name:adb-7338663619064447.7.privatelink.azuredatabricks.net Address: 100.64.254.16

You can see that it found the private IP for the workspace private endpoint on its own.

The challenge is if you are not using Azure DNS, or a properly configured forwarder, the hand off to the workspace URL is lost. For custom DNS you may need to add a CNAME that points to the workspace URL or an A record that points to the private endpoint IP address.