โ07-21-2024 09:21 AM - edited โ07-21-2024 09:23 AM
Hello,
Recently, when I create a new Databricks workspace on Azure, it automatically create a NAT Gateway which incurs additional cost !
When creating the workspace, I don't choose secure cluster connectivity, so I'm expecting not to have a NAT Gateway created.
Why NAT gateway is created for my Databricks workspaces knowing that it was not the case a couple weeks ago ? and how to delete it, or deploy workspaces without it ?
If this is not possible, how to avoid this extra cost related to NAT Gateway ?
โ07-21-2024 12:12 PM - edited โ07-21-2024 02:10 PM
Really weird, could you check how's your arm template looks like in rg deployments? To make sure that in your case there is no scc deployment?
I've created workspace with the same configuration and same region as your and my input for arm template looks like below. Inside managed rg I don't have NAT Gateway.
I've also created another workspace, but this time with SCC enabled. And now I've got NAT Gateway, just as is stated in documentation.
โ07-21-2024 10:49 AM - edited โ07-21-2024 11:02 AM
Hi @ziad ,
Do you create workspace with secure cluster connectivity? According to the documentation:
If you use secure cluster connectivity with the default VNet that Azure Databricks creates, Azure Databricks automatically creates a NAT gateway for outbound traffic from your workspaceโs subnets to the Azure backbone and public network. The NAT gateway is created within the managed resource group managed by Azure Databricks. You cannot modify this resource group or any resources provisioned within it.
The automatically-created NAT gateway incurs additional cost.
If you would like to avoid automatic creation of NAT gateway in secure cluster connectivity, you should perform VNET injection. Read below documentation entry for additional details:
Secure cluster connectivity - Azure Databricks | Microsoft Learn
Secure cluster connectivity - Azure Databricks | Microsoft Learn
EDIT: Sorry, didn't notice that you've mentioned you don't use secure cluster connectivity. So that's bit weird. Maybe at your organization there is some kind of azure policy that disables IP (so it enables implicitly SCC) ?
I've also created new workspace with the same configuration as yours and I don't have NAT gateway in DB manged RG
โ07-21-2024 11:21 AM - edited โ07-21-2024 11:26 AM
Thank you @szymon_dybczak for your reply.
Not only in my organization subscription, but also I've created a new Azure Free Tier account, and it automatically creates NAT Gateway for Databricks workspaces.
โ07-21-2024 12:12 PM - edited โ07-21-2024 02:10 PM
Really weird, could you check how's your arm template looks like in rg deployments? To make sure that in your case there is no scc deployment?
I've created workspace with the same configuration and same region as your and my input for arm template looks like below. Inside managed rg I don't have NAT Gateway.
I've also created another workspace, but this time with SCC enabled. And now I've got NAT Gateway, just as is stated in documentation.
โ07-21-2024 04:52 PM - edited โ07-21-2024 04:53 PM
Indeed weird! It seems that the issue has been resolved but I really don't understand what was going on. It seems it's an interface issue in Azure portal. For now, I'll go a head and accept your solution as it helps to better debug the issue. I will update the discussion in case I encountered the issue again. Thank you @szymon_dybczak for your help
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group