topic Re: NAT gateway with public IP for SCC disabled Databricks cluster in Administration & Architecture

NAT gateway with public IP for SCC disabled Databricks cluster

vsd — Wed, 27 Nov 2024 17:35:06 GMT

Hi Team,

We need to have single public IP for all outbound traffic flowing through our Databricks cluster. The Secure Cluster Connectivity (SCC) is disabled for our cluster and currently we get dynamic public IPs assigned to the VMs under managed resource group.

Can we explicitly configure NAT gateway (with a public IP) for a Databricks cluster having SCC disabled? Or does it have any dependency on SCC?

Thank you!

Re: NAT gateway with public IP for SCC disabled Databricks cluster

szymon_dybczak — Wed, 27 Nov 2024 18:05:43 GMT

Hi @vsd ,

I think so. If your workspace is VNET injected, then you can deploy your own NAT Gateway and redirect all outbound traffic from subnets through UDR to that gateway.

Re: NAT gateway with public IP for SCC disabled Databricks cluster

vsd — Thu, 28 Nov 2024 04:25:36 GMT

Thank you @szymon_dybczak! I was not able to find any reference documentation which mentions this very clearly. Do you have any reference for configuring such setup?

Re: NAT gateway with public IP for SCC disabled Databricks cluster

szymon_dybczak — Thu, 28 Nov 2024 06:46:37 GMT

Hi,

I don't think you will find any reference at MS docs. This is custom setup. But really similar approach is done in below knowledge base entry. In this article they choose to use firewall as stable egrees IP, but it doesn't matter. The same kind of steps you need to perform to make it work with your own NAT Gateway. Basically, you need to route all outbound traffic from databricks subnets to NAT Gateway

https://kb.databricks.com/cloud/azure-vnet-single-ip

Re: NAT gateway with public IP for SCC disabled Databricks cluster

vsd — Sat, 30 Nov 2024 15:36:37 GMT

Thank you @szymon_dybczak ! That's helpful!

Re: NAT gateway with public IP for SCC disabled Databricks cluster

szymon_dybczak — Sat, 30 Nov 2024 17:06:00 GMT

No problem @vsd, I'm glad that it help you 🙂