[Action required] [Azure] update outbound connectivity for classic workspaces

[AlexEsibov]

IMPORTANT NOTE: We have indefinitely delayed the automatic enforcement described below for workspaces that had enabled workspace IP access lists prior to July 29, 2024. We still recommend manually enforcing IP access lists on compute plane requests in these workspaces by taking the steps outlined below. 

Note: New IP access controls enabled on workspaces after July 29, 2024 are still enforced on data plane traffic, per the original communication below.

---------------------------------

Communication

To improve security for Azure Databricks customers, we’ll begin applying workspace IP access controls to compute plane traffic. This change will impact workspaces that use both secure cluster connectivity (no public IP) and workspace IP access lists. We’ll begin enforcing this change for all new workspaces starting July 29 2024 and all existing workspaces starting August 26 2024.

Required action

To ensure there’s no disruption to connectivity to the Azure Databricks control plane, you’ll need to take one of the following actions:

1. Add your compute plane IP addresses to the workspace IP access list.
2. Configure back-end private link for all workspaces.

If you are not the admin responsible for network connectivity to Azure Databricks, please forward this email to that person.  

Note that while this change only impacts secure cluster connectivity workspaces that use workspace IP access lists, Microsoft has announced that default outbound access for VMs in Azure will be retired on 30 September 2025. Therefore, we recommend proactively taking action.

Help and support 

If you have questions, get answers from community experts in Microsoft Q&A. If you have a support plan and you need technical help, open the Azure portal and select the question mark icon at the top of the page. 

Step-by-Step Instructions

Option 1) Add your compute plane IP addresses to the workspace IP access list

Note: If your compute plane traffic egresses through a firewall/proxy appliance, ensure that the IPs of the appliance are added to the workspace IP ACL policy. If it does not, read on for Azure NAT gateway deployment.

Note 2: Azure charges for Azure NAT Gateway. See pricing details here. 

1. Deploy one or more Azure NAT Gateways, if one doesn’t exist already
   1. How to check if Azure NAT Gateway already exists via Azure Portal
   1. Login to portal.azure.com
   2. Select the subscription that your workspace and resource group reside in
   3. Navigate to your Azure Databricks workspace
   4. Select the Resource Group that your workspace is in
   5. Check if there is a resource in your resource group of type “NAT Gateway” - if not, you do not have a NAT gateway
   2. How to check if Azure NAT Gateway already exists via via CLI (cloud shell)
      1.  Query the public subnet for an existing NAT Gateway

         az network vnet subnet show 
         --resource-group <resource group> 
         --vnet <vnet name> 
         --name <public subnet name> 
         --query "natGateway.id"

      2. Take the NAT gateway name from resource id from the previous step example and retrieve the public IP of the NAT gateway

         (/subscriptions/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/resourceGroups/brn/providers/Microsoft.Network/natGateways/[NAT_gateway_name])
         
         az network nat gateway show --resource-group <resource group> --name <nat-gateway name> --query publicIpAddresses[0].id
         

      3. Take the public IP from the above command example and confirm the public IP address is static

      4. (/subscriptions/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/resourceGroups/brn/providers/Microsoft.Network/publicIPAddresses/[NAT_gateway_name])) 
         
         az network public-ip show 
         --resource-group <resource group> 
         --name <public-ip name> 
         --query "{fqdn: dnsSettings.fqdn, address: ipAddress, type: publicIPAllocationMethod}"
         
         Example : az network public-ip show --resource-group brn --name [NAT_gateway_name]) --query "{fqdn: dnsSettings.fqdn, address: ipAddress, type: publicIPAllocationMethod}"
         {
           "address": "[IP_address]",
           "fqdn": null,
           "type": "[e.g., Static]"
         }
         

   3. How to create a NAT gateway
      1. Follow the steps outlined here to create a NAT gateway via UI or programatically: Manage a NAT gateway - Azure
   4. How to retrieve IPs for NAT gateway
      1. Via Azure Portal
         1. Login to portal.azure.com
         2. Select the subscription that your workspace and resource group reside in
         3. Navigate to your Azure Databricks workspace
         4. Select the Resource Group that your workspace is in
         5. Select the NAT gateway resource
         6. Navigate to “outbound IP”
         7. Copy the IP address 
         8. If there are multiple NAT gateways deployed (e.g., for multiple zones), collect all IP addresses for the NAT gateway 
      2. via CLI (cloud shell)

         az network public-ip show 
         --resource-group <resource group> 
         --name <public-ip name> 
         --query "{fqdn: dnsSettings.fqdn, address: ipAddress, type: publicIPAllocationMethod}"
         

2.  Add the Azure NAT Gateway IP addresses to the workspace IP access list
   1. Follow the steps outlined here to add the IP addresses for the NAT gateways collected above to your workspace IP ACL policy: https://learn.microsoft.com/en-us/azure/databricks/security/network/front-end/ip-access-list-workspa... 
3. Test that your deployment was successful
   1. Log in to your workspace
   2. Navigate to "Preview" > "View All" 
   3. Find "Enforce IP access list on Compute Plane Requests". On toggle on, IP ACL will be enforced on your NAT IP
   4. Wait for up to 10 minutes for the config to be applied to the workspace.
   5. Create and run a python notebook with a new cluster of any type except serverless.
      

      Cell #1 

      %pip install databricks-sdk --upgrade
      dbutils.library.restartPython()
      

      Cell #2

      from databricks.sdk import WorkspaceClient
      
      w = WorkspaceClient()
      w.clusters.list()
      

      If the code sample works, then your IP access list is set up correctly.
   6. In case of failures, toggle off "Enforce IP access list on Compute Plane Requests". Wait for up to 10 minutes for the config to be applied to the workspace.

4.  Optional - Use Azure virtual network service endpoints to access storage. To avoid using NAT for outbound connectivity for accessing storage, you can optionally deploy Azure virtual network service endpoints.

   1. In the Azure portal, go to the Databricks workspace object, click on “see more” and take note of the public subnet name. 

   2. Click on the Virtual network. open the public (host) subnet for you workspace and find the config entry “service endpoints”

   3. In the services drop down choose between “Micrososft.Storage” (for in region service endpoint networking) or “Microsoft.Storage.Global” (for cross region service endpoint networking)  Adding service endpoint networking for Databricks public subnet. Note: this approach has the following important limitations:

      1. Enabling service endpoints will change the route for all storage accounts accessed from that subnet, except routes using private endpoints. This means any routes configured to egress through, for example, a customer firewall, will be bypassed
      2. Each storage account must explicitly allow access from that public subnet. 

Option 2) Configure back-end private link for all workspaces, if not already done

1. Follow the steps outlined here to configure back-end private link for each workspace: Enable Azure Private Link back-end and front-end connections - Azure Databricks | Microsoft Learn 

Note: Azure charges for Azure Private Link. See details here.