topic Confusion about SCC, NAT, and Internet connection (Azure) in Administration & Architecture

Confusion about SCC, NAT, and Internet connection (Azure)

AlbertWang — Fri, 23 Aug 2024 04:52:30 GMT

Hi all,

I created an Azure Databricks workspace with the below features

Deployed in my Azure virtual network (VNet injection)
Enabled Secure cluster connectivity
Enabled the Back-end Private Link

I did not create and use a NAT. I found that I still can access Internet in a notebook with a compute.

Does it mean a NAT is not necessary in the architecture? My compute (virtual machine) does not have a public IP, how does the network traffic work from my compute to Internet?

The document Secure cluster connectivity says:

If you enable secure cluster connectivity on your workspace that uses VNet injection, Databricks recommends that your workspace has a stable egress public IP.

For deployments that need some customization, choose an Azure NAT gateway.

Does it mean the NAT is only necessary if I want my workspace has a stable egress public IP?

Thank you.

Re: Confusion about SCC, NAT, and Internet connection (Azure)

szymon_dybczak — Fri, 23 Aug 2024 06:20:08 GMT

Hi @AlbertWang ,

If you choose SCC option, there shouldn't be NAT gateway created. You can check that yourself if you go the managed resource group that is created when you deploying databricks workspace.

When you use SCC then if you would like to have outbound traffic you need to use your VNET's egress solution.
Depending on you setting it could be for instance:

- Azure Firewall with allows outbound rules configured. (traffic from databricks subnet can routed using route table to this firewall)

- Load Balancer with Outbound NAT rules or NAT Gateway linked to both workspace subnets.

Re: Confusion about SCC, NAT, and Internet connection (Azure)

AlbertWang — Fri, 23 Aug 2024 20:47:06 GMT

Hi @szymon_dybczak Thank you for your reply.

Do you mean "If you choose SCC option, there should be NAT gateway created"?

I checked there was no NAT created. Moreover, I have no Azure Firewall, Load Balancer, and NAT gateway, but my compute still can connect Internet.

Re: Confusion about SCC, NAT, and Internet connection (Azure)

szymon_dybczak — Fri, 23 Aug 2024 22:12:45 GMT

Hi @AlbertWang,

I mean there shouldn't be nat gateway created automatically when you choose SCC + vnet injection.

I totally forgot about azure default outbound access. In Azure, virtual machines created in a virtual network without explicit outbound connectivity defined are assigned a default outbound public IP address. So that's why you outbound access to internet.

But take into account that in near future this feature would be retired from azure:

https://learn.microsoft.com/en-us/azure/virtual-network/ip-services/default-outbound-access