This is a classic hub-spoke + on-premises hybrid networking scenario. Here's how to architect it end-to-end.

Architecture Overview

The traffic flow will be:

VM (VNet-App) --> ExpressRoute/VPN Gateway --> On-Prem Proxy Server --> ExpressRoute/VPN Gateway --> VNet-PE-ENDPOINT --> Private Endpoint --> Azure Databricks

Step 1: Network Connectivity Between VNets and On-Premises

You need two connectivity paths -- both going through your on-premises network:

VM VNet (VNet-App) to On-Premises:

• Configure an ExpressRoute circuit or Site-to-Site VPN Gateway in VNet-App (or a hub VNet peered to VNet-App)
• This allows the VM to route traffic to the on-premises proxy

On-Premises to Private Endpoint VNet (VNet-PE-ENDPOINT):

• Configure an ExpressRoute circuit or Site-to-Site VPN Gateway in VNet-PE-ENDPOINT (or a hub VNet peered to it)
• This allows the on-premises proxy to reach the private endpoint's private IP

Recommended: Hub-Spoke Topology

Rather than connecting each VNet individually, use a hub VNet with a single ExpressRoute/VPN gateway:

VNet-App (spoke) ---peering---> Hub VNet <---peering--- VNet-PE-ENDPOINT (spoke)

                                   |

                            ExpressRoute/VPN

                                   |

                           On-Premises Network

                           (Proxy Server here)

Enable "Allow Gateway Transit" on the hub peering and "Use Remote Gateway" on each spoke peering so all spokes can use the hub's gateway.

Step 2: Configure the On-Premises Proxy Server

The proxy server (e.g., Squid, nginx, or an enterprise proxy like Zscaler/Blue Coat) must be configured to:

Allow HTTPS traffic to Databricks endpoints:

• Your workspace URL: adb-xxxxxxxxxxxx.xx.azuredatabricks.net (port 443)
• Browser auth URL: region.pl-auth.azuredatabricks.net (port 443)
• Additional ports if needed: 6666, 3306, 8443-8451

Forward traffic toward the Azure private endpoint IP:

• The proxy must resolve the Databricks workspace URL to the private IP of the private endpoint (not the public IP)
• This requires proper DNS configuration (see Step 3)

Proxy configuration example (Squid):

# Allow Databricks workspace traffic

acl databricks_hosts dstdomain .azuredatabricks.net

http_access allow databricks_hosts

Step 3: DNS Configuration (Critical)

This is the most important step. The proxy server must resolve Databricks URLs to private IPs, not public IPs.

Option A: Conditional DNS Forwarding (Recommended)

On your on-premises DNS server, configure a conditional forwarder:

• Zone: privatelink.azuredatabricks.net
• Forward to: An Azure DNS Forwarder (a VM or Azure Firewall DNS proxy in your hub VNet)

Important: Do NOT forward directly to 168.63.129.16 -- this Azure DNS IP only responds to queries from within Azure VNets. You need an intermediary forwarder.

Azure DNS Forwarder options:

• A small Windows/Linux VM running DNS forwarding (e.g., BIND, Windows DNS, dnsmasq)
• Azure Firewall with DNS proxy enabled
• Azure DNS Private Resolver (managed service, no VM needed)

Option B: Manual DNS A Records

If conditional forwarding isn't possible, create static A records on your on-premises DNS:

adb-xxxxxxxxxxxx.xx.azuredatabricks.net  -->  10.x.x.x  (private endpoint IP)

region.pl-auth.azuredatabricks.net       -->  10.x.x.x  (same private endpoint IP)

Find the private IP from: Azure Portal > Private Endpoint > Network Interface > IP Configuration

Note: Do NOT override accounts.azuredatabricks.net -- the Account Console must resolve publicly.

Step 4: VM Proxy Configuration

Configure the VM in VNet-App to route Databricks traffic through the on-premises proxy:

Environment variables (Linux):

export HTTPS_PROXY=http://proxy.onprem.company.com:8080

export HTTP_PROXY=http://proxy.onprem.company.com:8080

export NO_PROXY=169.254.169.254,168.63.129.16

For Databricks CLI or API calls:

export HTTPS_PROXY=http://proxy.onprem.company.com:8080

databricks clusters list --profile my-workspace

For application code (Python example):

import os

os.environ['HTTPS_PROXY'] = 'http://proxy.onprem.company.com:8080'

Step 5: NSG and Firewall Rules

Ensure Network Security Groups allow the required traffic:

VNet-App NSG (outbound):

• Allow TCP 443 outbound to on-premises proxy IP

On-premises firewall:

• Allow the proxy to reach VNet-PE-ENDPOINT subnet on TCP 443

VNet-PE-ENDPOINT NSG (inbound to private endpoint subnet):

• Allow TCP 443, 6666, 3306, 8443-8451 from on-premises network CIDR

Step 6: Verify End-to-End Connectivity

From the on-premises proxy server:

nslookup adb-xxxxxxxxxxxx.xx.azuredatabricks.net

# Should resolve to the private IP (e.g., 10.x.x.x)

From the VM (through proxy):

curl -x http://proxy.onprem.company.com:8080 \

  https://adb-xxxxxxxxxxxx.xx.azuredatabricks.net/api/2.0/clusters/list \

  -H "Authorization: Bearer <token>"

Summary Checklist

Key Gotcha

The most common failure is DNS resolution. If the proxy resolves the Databricks URL to a public IP instead of the private endpoint IP, the connection will fail because public access is disabled. Always verify with nslookup from the proxy server itself.

References

• Configure Inbound Private Link -- Azure Databricks
• Azure Private Link Concepts -- Databricks
• Resolve Databricks Private Endpoint URLs from On-Prem -- Microsoft Q&A
• Hub-Spoke Network Topology in Azure