Creating Azure Databricks Workspace Without NAT Gateway

[ziad]

Hello,

Recently, when I create a new Databricks workspace on Azure, it automatically create a NAT Gateway which incurs additional cost !

When creating the workspace, I don't choose secure cluster connectivity, so I'm expecting not to have a NAT Gateway created.

Why NAT gateway is created for my Databricks workspaces knowing that it was not the case a couple weeks ago ? and how to delete it, or deploy workspaces without it ?

If this is not possible, how to avoid this extra cost related to NAT Gateway ?

[szymon_dybczak]

Hi @ziad ,

Do you create workspace with secure cluster connectivity? According to the documentation: 

If you use secure cluster connectivity with the default VNet that Azure Databricks creates, Azure Databricks automatically creates a NAT gateway for outbound traffic from your workspace’s subnets to the Azure backbone and public network. The NAT gateway is created within the managed resource group managed by Azure Databricks. You cannot modify this resource group or any resources provisioned within it.

The automatically-created NAT gateway incurs additional cost.

If you would like to avoid automatic creation of NAT gateway in secure cluster connectivity, you should perform VNET injection. Read below documentation entry for additional details:

Secure cluster connectivity - Azure Databricks | Microsoft Learn

Secure cluster connectivity - Azure Databricks | Microsoft Learn

EDIT: Sorry, didn't notice that you've mentioned you don't use secure cluster connectivity. So that's bit weird. Maybe at your organization there is some kind of azure policy that disables IP (so it enables implicitly SCC) ?

I've also created new workspace with the same configuration as yours and I don't have NAT gateway in DB manged RG

[ziad]

Thank you @szymon_dybczak for your reply.

Not only in my organization subscription, but also I've created a new Azure Free Tier account, and it automatically creates NAT Gateway for Databricks workspaces.

[szymon_dybczak · Accepted solution]

Really weird, could you check how's your arm template looks like in rg deployments? To make sure that in your case there is no scc deployment?
I've created workspace with the same configuration and same region as your and my input for arm template looks like below. Inside managed rg I don't have NAT Gateway.

I've also created another workspace, but this time with SCC enabled. And now I've got NAT Gateway, just as is stated in documentation.

[ziad]

Indeed weird! It seems that the issue has been resolved but I really don't understand what was going on. It seems it's an interface issue in Azure portal. For now, I'll go a head and accept your solution as it helps to better debug the issue. I will update the discussion in case I encountered the issue again. Thank you @szymon_dybczak for your help