[Action required] [Azure] update outbound connectivity for classic workspaces

[AlexEsibov]

Communication

To improve security for Azure Databricks customers, we’ll begin applying workspace IP access controls to compute plane traffic. This change will impact workspaces that use both secure cluster connectivity (no public IP) and workspace IP access lists. We’ll begin enforcing this change for all new workspaces starting July 29 2024 and all existing workspaces starting August 26 2024.

Required action

To ensure there’s no disruption to connectivity to the Azure Databricks control plane, you’ll need to take one of the following actions:

1. Add your compute plane IP addresses to the workspace IP access list.
2. Configure back-end private link for all workspaces.

If you are not the admin responsible for network connectivity to Azure Databricks, please forward this email to that person.  

Note that while this change only impacts secure cluster connectivity workspaces that use workspace IP access lists, Microsoft has announced that default outbound access for VMs in Azure will be retired on 30 September 2025. Therefore, we recommend proactively taking action.

Help and support 

If you have questions, get answers from community experts in Microsoft Q&A. If you have a support plan and you need technical help, open the Azure portal and select the question mark icon at the top of the page. 

Step-by-Step Instructions

Option 1) Add your compute plane IP addresses to the workspace IP access list

Note: If your compute plane traffic egresses through a firewall/proxy appliance, ensure that the IPs of the appliance are added to the workspace IP ACL policy. If it does not, read on for Azure NAT gateway deployment.

Note 2: Azure charges for Azure NAT Gateway. See pricing details here. 

1. Deploy one or more Azure NAT Gateways, if one doesn’t exist already
   1. How to check if Azure NAT Gateway already exists via Azure Portal
   1. Login to portal.azure.com
   2. Select the subscription that your workspace and resource group reside in
   3. Navigate to your Azure Databricks workspace
   4. Select the Resource Group that your workspace is in
   5. Check if there is a resource in your resource group of type “NAT Gateway” - if not, you do not have a NAT gateway
   2. How to check if Azure NAT Gateway already exists via via CLI (cloud shell)
      1.  Query the public subnet for an existing NAT Gateway

         az network vnet subnet show 
         --resource-group <resource group> 
         --vnet <vnet name> 
         --name <public subnet name> 
         --query "natGateway.id"

      2. Take the NAT gateway name from resource id from the previous step example and retrieve the public IP of the NAT gateway

         (/subscriptions/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/resourceGroups/brn/providers/Microsoft.Network/natGateways/[NAT_gateway_name])
         
         az network nat gateway show --resource-group <resource group> --name <nat-gateway name> --query publicIpAddresses[0].id
         

      3. Take the public IP from the above command example and confirm the public IP address is static

      4. (/subscriptions/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/resourceGroups/brn/providers/Microsoft.Network/publicIPAddresses/[NAT_gateway_name])) 
         
         az network public-ip show 
         --resource-group <resource group> 
         --name <public-ip name> 
         --query "{fqdn: dnsSettings.fqdn, address: ipAddress, type: publicIPAllocationMethod}"
         
         Example : az network public-ip show --resource-group brn --name [NAT_gateway_name]) --query "{fqdn: dnsSettings.fqdn, address: ipAddress, type: publicIPAllocationMethod}"
         {
           "address": "[IP_address]",
           "fqdn": null,
           "type": "[e.g., Static]"
         }
         

   3. How to create a NAT gateway
      1. Follow the steps outlined here to create a NAT gateway via UI or programatically: Manage a NAT gateway - Azure
   4. How to retrieve IPs for NAT gateway
      1. Via Azure Portal
         1. Login to portal.azure.com
         2. Select the subscription that your workspace and resource group reside in
         3. Navigate to your Azure Databricks workspace
         4. Select the Resource Group that your workspace is in
         5. Select the NAT gateway resource
         6. Navigate to “outbound IP”
         7. Copy the IP address 
         8. If there are multiple NAT gateways deployed (e.g., for multiple zones), collect all IP addresses for the NAT gateway 
      2. via CLI (cloud shell)

         az network public-ip show 
         --resource-group <resource group> 
         --name <public-ip name> 
         --query "{fqdn: dnsSettings.fqdn, address: ipAddress, type: publicIPAllocationMethod}"
         

2.  Add the Azure NAT Gateway IP addresses to the workspace IP access list
   1. Follow the steps outlined here to add the IP addresses for the NAT gateways collected above to your workspace IP ACL policy: https://learn.microsoft.com/en-us/azure/databricks/security/network/front-end/ip-access-list-workspa... 
3. Test that your deployment was successful
   1. Log in to your workspace
   2. Navigate to "Preview" > "View All" 
   3. Find "Enforce IP access list on Compute Plane Requests". On toggle on, IP ACL will be enforced on your NAT IP
   4. Wait for up to 10 minutes for the config to be applied to the workspace.
   5. Create and run a python notebook with a new cluster of any type except serverless.
      

      Cell #1 

      %pip install databricks-sdk --upgrade
      dbutils.library.restartPython()
      

      Cell #2

      from databricks.sdk import WorkspaceClient
      
      w = WorkspaceClient()
      w.clusters.list()
      

      If the code sample works, then your IP access list is set up correctly.
   6. In case of failures, toggle off "Enforce IP access list on Compute Plane Requests". Wait for up to 10 minutes for the config to be applied to the workspace.

4.  Optional - Use Azure virtual network service endpoints to access storage. To avoid using NAT for outbound connectivity for accessing storage, you can optionally deploy Azure virtual network service endpoints.

   1. In the Azure portal, go to the Databricks workspace object, click on “see more” and take note of the public subnet name. 

   2. Click on the Virtual network. open the public (host) subnet for you workspace and find the config entry “service endpoints”

   3. In the services drop down choose between “Micrososft.Storage” (for in region service endpoint networking) or “Microsoft.Storage.Global” (for cross region service endpoint networking)  Adding service endpoint networking for Databricks public subnet. Note: this approach has the following important limitations:

      1. Enabling service endpoints will change the route for all storage accounts accessed from that subnet, except routes using private endpoints. This means any routes configured to egress through, for example, a customer firewall, will be bypassed
      2. Each storage account must explicitly allow access from that public subnet. 

Option 2) Configure back-end private link for all workspaces, if not already done

1. Follow the steps outlined here to configure back-end private link for each workspace: Enable Azure Private Link back-end and front-end connections - Azure Databricks | Microsoft Learn 

Note: Azure charges for Azure Private Link. See details here. 

[OvZ]

Hi, is there a place in the databricks logs where I can see if this switch is enabled (other then in de ui)>> Enforce IP ACL on Dataplane Requests ? and second databricks uses NAT or private link connection ?  thx in adv, Oscar

[Said]

I have a doubt on this. If i routed databricks traffic through Azure firewall, whether i need to whitelist the Firewall public IP or is there anything else i have to consider.

[AlexEsibov]

@Said good question - you should allow-list the public IPs associated with the outbound traffic from the Databricks compute plane. Depending on the specific proxy/firewall appliance and configuration, some will obfuscate the source IPs, while others can preserve them. If traffic egressing from your Azure firewall uses the public IP of the firewall, I would make sure to allowlist that IP in the workspace IP ACL.

[chang_vnext]

Hi, say for example, we have two types of databricks workspace being used. The configurations are

1. Premium tier, with

    - IP access list enabled

    - Public Network access enabled

    - Required NSG Rules: All Rules

    - Private endpoint: ui_api service only

2. Standard tier, with

    - IP access list disabled

    - Public Network access enabled

    - Required NSG Rules: All Rules

With these two configurations, do we still need to do anything to ensure the these two workspaces are still working as expected?

[yankaiZhang-db]

@chang_vnext Hi, only the (1) bucket will potentially be affected. Please check the "enableNoPublicIp" parameter of your Azure Databricks workspace. If for your premium tier workspace (1) you have enableNoPublicIp = true, then this workspace is affected and please add the public IP of your NAT gateway to your workspace IP access list, then turn on "Enforce IP access list on Compute Plane Requests" to secure your workspace.

[AlexEsibov]

@OvZ sorry I missed your questions:

1) To check if the switch is enabled, you could check logs but using a service 'workspace', action of 'setSetting' and tags.settingTypeName of 'enforce_ip_acl_on_dp'. 

2) Your classic compute resources can connect to the Azure Databricks Control Plane over NAT or Private Link - it depends on your deployment. 

[elvisleung]

Is there any way to check what IP is used from the compute plane to control plane, for example, logs?

In my case, the traffic going out from the compute plane SNet is not going through NAT gateway or firewall. I believe that some sort of Azure public IP will be used. I would like to confirm what are the exact IP(s) that I need to whitelist.

The workspace is with SCC and IP whitelist on and without backend private endpoint.

[Said]

I understand that the change will primarily affect workspaces utilizing both secure cluster connectivity (no public IP) and workspace IP access lists. In my Azure Databricks workspace, I'm currently only using secure cluster connectivity, and I do not have workspace IP access lists configured.

My question is: Will this upcoming change regarding workspace IP access controls impact my environment in any way?

[AlexEsibov]

@Said this change will not impact you. You must meet both conditions as you pointed out. 

[rugger-bricks]

There is a simple way to check your outbound IP from the Databricks compute plane - its a great way to validate a NAT gateway or FW NAT IP .. also can tell you the default SNAT ip if you are NPIP and going direct to the internet. 

In a notebook cell run : 

%sh
curl ipinfo.io | grep ip