[Action required] [AWS] update outbound connectivity for classic workspaces

[AlexEsibov]

IMPORTANT NOTE: We have indefinitely delayed the automatic enforcement described below for workspaces that had enabled workspace IP access lists prior to July 29, 2024. We still recommend manually enforcing IP access lists on compute plane requests in these workspaces by taking the steps outlined below. 

Note: New IP access controls enabled on workspaces after July 29, 2024 are still enforced on data plane traffic, per the original communication below.

---------------------------------

Communication

To enhance security, we are making a change to workspaces that use the workspace IP access lists feature. For these workspaces, we will begin to apply workspace IP access controls to compute plane traffic. See Action Required and Timeline below for details.

Action Required

This change will impact all new workspaces on July 29 2024, and existing workspaces on August 26 2024. 

To ensure your compute plane can continue to talk to the Databricks control plane - take action to add the NAT gateway IP addresses to your workspace IP access list. Step-by-step instructions are available below.

Timeline

The required actions must be taken by the following dates:

• Starting on July 29 2024, all new workspaces that use workspace IP access lists will begin enforcing workspace IP access lists on compute plane traffic
• By August 26 2024, all existing workspaces that use workspace IP access lists will begin enforcing workspace IP access lists on compute plane traffic

Step-by-Step Instructions

Note: If your compute plane traffic egresses through a firewall/proxy appliance, ensure that the IPs of the appliance are added to the workspace IP ACL policy. If it does not, read on for NAT gateway deployment.

1. Retrieve IPs for the NAT gateway (via AWS console)
1. Find your NAT gateway in
   https://<REGION>.console.aws.amazon.com/vpcconsole/home?region=<REGION>#NatGateways:
   You can identify the NAT gateway by filtering by your compute plane VPC ID.
2. Note down "Primary public IPv4 address". That is your public IP of your NAT gateway.
2. Adding the NAT Gateway IP addresses to the workspace IP access list
1. Follow the steps outlined here to add the IP addresses for the NAT gateways collected above to your workspace IP ACL policy:
   https://docs.databricks.com/en/security/network/front-end/ip-access-list-workspace.html
3. Test that your deployment was successful 
1. Log in to your workspace
2. Navigate to "Preview" > "View All" 
3. Find "Enforce IP access list on Compute Plane Requests". On toggle on, IP ACL will be enforced on your NAT IP
4. Wait for up to 10 minutes for the config to be applied to the workspace.
5. Create and run a python notebook with a new cluster of any type except serverless.
   

   Cell #1 

   %pip install databricks-sdk --upgrade
   dbutils.library.restartPython()
   

   Cell #2

   from databricks.sdk import WorkspaceClient
   
   w = WorkspaceClient()
   w.clusters.list()
   

   If the code sample works, then your IP access list is set up correctly.
6. In case of failures, toggle off "Enforce IP access list on Compute Plane Requests". Wait for up to 10 minutes for the config to be applied to the workspace.