topic Re: Different NCC having same subnets in Administration & Architecture

Different NCC having same subnets

loic — Wed, 29 Jan 2025 09:43:48 GMT

Hello,

We are forwarding this Microsoft tutorial to secure our storage access:

https://learn.microsoft.com/en-us/azure/databricks/security/network/serverless-network-security/serverless-firewall

We have a weird behavior when we create several NCCs in the same region.
Indeed, it seems those NCCs have the same subnets.
When we request network-connectivity-configs endpoint, we have 10 subnets, this is the kind of subnets that are listed:

"/subscriptions/XXXXXXXXXX/resourceGroups/prod-francecentral-snp-1-compute-4/providers/Microsoft.Network/virtualNetworks/prod-francecentral-snp-1-compute-4/subnets/worker-subnet",

This is a single entry example, we have exactly the same entries for both NCCs!
Thus, when we authorize subnets from NCC1 to get access to the storage, then, a Warehouse serverless query that is done from a Databricks workspace binded to NCC2 is able to connect to this storage!

Does somebody can tell me what I am doing wrong?
I know that serverless was introduced recently in France, maybe there is an issue here?

Regards,
Loïc

Re: Different NCC having same subnets

Rjdudley — Wed, 29 Jan 2025 14:02:19 GMT

I think this is the expected behavior, and you don't need multiple NCCs to the same VNet. Remember that serverless compute is in pre-warmed pools, just waiting for action. These are large pools with thousands of nodes, used by many customers connecting to the pool. They don't create new subnets for every NCC. Although serverless nodes used by two customers can exist in the same subnet at the same time, there are layers of isolation to prevent cross-talk between nodes.

Re: Different NCC having same subnets

loic — Mon, 03 Feb 2025 14:29:41 GMT

"They don't create new subnets for every NCC. "
That's indeed what I observe.
Maybe my issue is more that the REST API that I use:
https://docs.databricks.com/api/account/networkconnectivity/getnetworkconnectivityconfiguration

doesn't return the stable IPs that I should add to my storage firewall. Instead, it only returns subnets (same for all NCC).
Thus, since Serverless of the workspace binded to NCC2 use same subnets that workspace binded to NCC1, I can not do "NCC per environment" pattern as described on this site:
https://medium.com/databricks-platform-sme/azure-databricks-serverless-ncc-design-considerations-patterns-ff9c61bfb8bd

But anyhow, finally, we are going to keep the storage public for now (no firewall) since there is too much constrain to share data. So I am not going to use NCC for the moment.

Re: Different NCC having same subnets

Rjdudley — Mon, 03 Feb 2025 19:14:42 GMT

Leaving your storage wide open is a horrible idea. That is how data breaches happen and the penalties are becoming more severe.

I think you're trying to set this up incorrectly. You don't need to know the IP range because you don't add IP ranges to your firewall, you add the subnets using the virtual networks block.

Re: Different NCC having same subnets

loic — Thu, 06 Feb 2025 10:16:31 GMT

Ok, so no, I correctly set the subnets of my NCC in the Virtual Networks setting as documented:
https://learn.microsoft.com/en-us/azure/databricks/security/network/serverless-network-security/serverless-firewall

This setting is working fine, without this, I was not able to do SQL Warehouse serverless requests to my storage.
My original question was about the fact that I was also able to do SQL Warehouse serverless requests from a workspace binded to NCC-2 meanwhile my storage was configured with the list of subnets from NCC-1.
@Rjdudley , if according to you, the expected behavior is: "They don't create new subnets for every NCC.", then, I have to understand that what I observed is normal.
Thanks you help