Hi Databricks Community,
I'm working through some networking challenges when connecting Databricks clusters to various data sources and wanted to get advice or best practices from others who may have faced similar issues.
Current Setup:
I have four types of source systems that I need to connect to from Databricks:
1. Customer Plane Clusters → Source in Azure VNet
Approach: Peered the Databricks Customer VNet with the source system’s VNet.
Connectivity: Whitelisted the NAT Gateway Public IP in the source system’s firewall.
2. Customer Plane Clusters → On-Prem System
Approach: Established a Site-to-Site VPN between the Databricks Customer VNet and On-Prem network.
Connectivity: Whitelisted the private IPs on the on-prem side.
3. Control Plane Clusters (Serverless Compute) → Azure Services (Storage Account, MySQL, etc.)
Approach: Using Network Connectivity Configuration (NCC) in Databricks.
4. Control Plane Clusters (Serverless Compute) → On-Prem System
Approach: Not applicable yet — looking for guidance here.
Connectivity challenge: Unable to establish direct connectivity due to lack of support for peering or site-to-site connections from Control Plane to On-Prem.
The Problem: However, I'm running into networking limitations when trying to connect Serverless Compute (Control Plane) to systems behind firewalls - especially in on-premises or other CSPs / SaaS applications.
Issue A: No Static Outbound IPs for Serverless Compute
For external systems behind a firewall, there is no static public IP address available from serverless compute to whitelist.
Issue B: No Network-Level Integration with On-Prem Networks
Unlike customer-managed clusters, serverless compute does not support peering or site-to-site connections, and there is no direct network-level communication.
Issue C: Limited Support for Hybrid or Multi-Cloud Scenarios
There’s currently no supported way to securely connect Databricks serverless compute to:
- On-prem systems
- Other cloud providers (AWS/GCP)
- Third-party SaaS applications requiring IP-based access control
