Greetings @chandru44 , Thanks for sharing this detailed networking setup—you've clearly done thorough work mapping out your connectivity patterns. You've correctly identified the fundamental architectural limitation with serverless compute and on-premises connectivity. Let me address your concerns and provide some practical guidance.

Understanding the Limitation

You're absolutely right that Databricks serverless compute does not currently support direct network-level integration with on-premises systems. This is by design—serverless compute runs in the Databricks-managed control plane, not in your customer VNet, which means traditional networking approaches (VNet peering, Site-to-Site VPN) cannot be applied.

Why NCC Doesn't Solve On-Prem Connectivity

Network Connectivity Configurations (NCC) are designed specifically for Azure-native resources only (Storage Accounts, MySQL, Cosmos DB, etc.) using Azure Private Link and managed private endpoints. NCC cannot establish connectivity to on-premises systems because it relies on Azure's Private Link infrastructure, which doesn't extend to on-prem networks.

Recommended Workarounds

Option 1: Hybrid Data Sync Pattern (Recommended for Most Cases)

The most reliable approach is to use a staging/synchronization layer:

1. Use your customer plane clusters (with Site-to-Site VPN connectivity) to extract data from on-prem systems
2. Land the data in Azure storage (ADLS Gen2, Blob Storage) with appropriate governance
3. Use serverless compute to process the data in Azure
4. Optionally sync processed results back to on-prem via customer plane clusters

This pattern prevents overwhelming on-prem systems with cloud-scale operations and provides better performance and cost optimization.

Option 2: Expose On-Prem Services via Azure Gateway

If real-time access is required:

1. Deploy an Azure Application Gateway or API Management service in your Azure VNet
2. Configure ExpressRoute or Site-to-Site VPN from Azure to on-prem
3. Expose on-prem services through this gateway with appropriate authentication
4. Configure NCC firewall rules to allow serverless compute to access the gateway's subnet

Note: This still requires the gateway to be an Azure resource that NCC can reach.

Option 3: Use Classic Compute for On-Prem Connectivity

For workloads that must have direct on-prem connectivity, continue using customer plane clusters with VNet injection. You can design a hybrid architecture where:

- Serverless compute handles Azure-native workloads (better performance, lower cost)
- Classic compute clusters handle on-prem connectivity workloads
- Both layers interact via Unity Catalog tables

Addressing Your Specific Issues

Issue A - No Static Outbound IPs: Correct—serverless compute does not provide static public IPs for whitelisting. For customer plane clusters, you can whitelist NAT Gateway IPs, but this isn't available for serverless.

Issue B - No Network Integration: This is a fundamental architectural constraint. Serverless compute prioritizes rapid scaling and zero infrastructure management over custom networking.

Issue C - Multi-Cloud/SaaS Limitations: For external systems requiring IP whitelisting, the hybrid sync pattern (Option 1) or exposing services via Azure-native endpoints (Option 2) are your best options.

Additional Considerations

- Delta Sharing: For governed data distribution between on-prem and cloud environments, consider using Delta Sharing to share curated datasets without data duplication.
- Cost Optimization: The hybrid approach actually helps with cost—you avoid unnecessary data egress charges and can optimize compute usage based on workload type.
- Security: The staging pattern provides better data governance and audit trails compared to direct connectivity.

Unfortunately, there's no direct solution for serverless-to-on-prem connectivity in the current architecture. The workarounds above represent the practical approaches used by organizations facing similar constraints.

Hope this helps clarify your options!

Cheers, Louis.