cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Confusion about SCC, NAT, and Internet connection (Azure)

AlbertWang
Contributor

Hi all,

I created an Azure Databricks workspace with the below features

I did not create and use a NAT. I found that I still can access Internet in a notebook with a compute.

Does it mean a NAT is not necessary in the architecture? My compute (virtual machine) does not have a public IP, how does the network traffic work from my compute to Internet?

The document Secure cluster connectivity says:

If you enable secure cluster connectivity on your workspace that uses VNet injection, Databricks recommends that your workspace has a stable egress public IP.

  • For deployments that need some customization, choose an Azure NAT gateway.

Does it mean the NAT is only necessary if I want my workspace has a stable egress public IP?

Thank you.

1 ACCEPTED SOLUTION

Accepted Solutions

Hi @AlbertWang,

I mean there shouldn't be nat gateway created automatically when you choose SCC + vnet injection.

I totally forgot about azure default outbound access. In Azure, virtual machines created in a virtual network without explicit outbound connectivity defined are assigned a default outbound public IP address. So that's why you outbound access to internet.

But take into account that in near future this feature would be retired from azure: 

 

 

https://learn.microsoft.com/en-us/azure/virtual-network/ip-services/default-outbound-access

View solution in original post

3 REPLIES 3

szymon_dybczak
Contributor III

Hi @AlbertWang ,

If you choose SCC option, there shouldn't be NAT gateway created. You can check that yourself if you go the managed resource group that is created when you deploying databricks workspace.

When you use SCC then if you would like to have outbound traffic you need to use your VNET's egress solution.
Depending on you setting it could be for instance:

- Azure Firewall with allows outbound rules configured. (traffic from databricks subnet can routed using route table to this firewall)

- Load Balancer with Outbound NAT rules or NAT Gateway linked to both workspace subnets.

 

Hi @szymon_dybczak Thank you for your reply.

Do you mean "If you choose SCC option, there should be NAT gateway created"?

I checked there was no NAT created. Moreover, I have no Azure Firewall, Load Balancer, and NAT gateway, but my compute still can connect Internet.

 

Hi @AlbertWang,

I mean there shouldn't be nat gateway created automatically when you choose SCC + vnet injection.

I totally forgot about azure default outbound access. In Azure, virtual machines created in a virtual network without explicit outbound connectivity defined are assigned a default outbound public IP address. So that's why you outbound access to internet.

But take into account that in near future this feature would be retired from azure: 

 

 

https://learn.microsoft.com/en-us/azure/virtual-network/ip-services/default-outbound-access

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group