topic Re: NAT Gateway with Azure Databricks in Administration & Architecture

NAT Gateway with Azure Databricks

ph1l0s0ph3r — Wed, 11 Sep 2024 13:13:53 GMT

Hi all, what are the steps to enable external communication through a NAT Gateway? Our Databricks instance was created through VNET injection and did not have a NAT gateway created by default. We now want to pass external traffic through NAT gateway rather than the Databricks IP. Thanks

Re: NAT Gateway with Azure Databricks

szymon_dybczak — Wed, 11 Sep 2024 13:51:55 GMT

Hi @ph1l0s0ph3r ,

NAT Gateway is created by default only if you use SCC with default VNet that Azure Databricks creates. Look at below entry in documentation:

- Secure cluster connectivity - Azure Databricks | Microsoft Learn

In you case, when you have deployed Databricks instance into your own VNet you need to configure NAT Gateway by yourself.
Basically, you have following option to configure egress with NPIP in VNet Injected workspace:

Egress Load Balancer
Azure NAT Gateway
Azure Firewall

You're interested in option 2 - Azure NAT Gateway. So to set it up you need to create:

- NAT Gateway in Azure

- configure the gateway on both of the workspace’s subnets to ensure that all outbound traffic to the Azure backbone and public network transits through it.

Re: NAT Gateway with Azure Databricks

ph1l0s0ph3r — Wed, 11 Sep 2024 14:04:44 GMT

Thank you, @szymon_dybczak ! I will try this and get back on this thread.

Is a route table necessary for allowing outbound traffic in the subnets at all? Or will just adding the NAT gateway to both the subnets be sufficient?

Re: NAT Gateway with Azure Databricks

szymon_dybczak — Wed, 11 Sep 2024 14:21:15 GMT

Adding the NAT Gateway to both subnet should be sufficient. If you encounter any problem let us know 😉

Re: NAT Gateway with Azure Databricks

ph1l0s0ph3r — Wed, 11 Sep 2024 15:05:18 GMT

I made the changes and it works! I was able to communicate to the NAT Gateway IP in our external on-prem application.

I noticed though that Databricks instance is not showing the NAT Gateway within Parameters. Is this an issue?