[Action required] [AWS] update outbound connectivity for classic workspaces

[AlexEsibov]

IMPORTANT NOTE: We have indefinitely delayed the automatic enforcement described below for workspaces that had enabled workspace IP access lists prior to July 29, 2024. We still recommend manually enforcing IP access lists on compute plane requests in these workspaces by taking the steps outlined below. 

Note: New IP access controls enabled on workspaces after July 29, 2024 are still enforced on data plane traffic, per the original communication below.

---------------------------------

Communication

To enhance security, we are making a change to workspaces that use the workspace IP access lists feature. For these workspaces, we will begin to apply workspace IP access controls to compute plane traffic. See Action Required and Timeline below for details.

Action Required

This change will impact all new workspaces on July 29 2024, and existing workspaces on August 26 2024. 

To ensure your compute plane can continue to talk to the Databricks control plane - take action to add the NAT gateway IP addresses to your workspace IP access list. Step-by-step instructions are available below.

Timeline

The required actions must be taken by the following dates:

• Starting on July 29 2024, all new workspaces that use workspace IP access lists will begin enforcing workspace IP access lists on compute plane traffic
• By August 26 2024, all existing workspaces that use workspace IP access lists will begin enforcing workspace IP access lists on compute plane traffic

Step-by-Step Instructions

Note: If your compute plane traffic egresses through a firewall/proxy appliance, ensure that the IPs of the appliance are added to the workspace IP ACL policy. If it does not, read on for NAT gateway deployment.

1. Retrieve IPs for the NAT gateway (via AWS console)
1. Find your NAT gateway in
   https://<REGION>.console.aws.amazon.com/vpcconsole/home?region=<REGION>#NatGateways:
   You can identify the NAT gateway by filtering by your compute plane VPC ID.
2. Note down "Primary public IPv4 address". That is your public IP of your NAT gateway.
2. Adding the NAT Gateway IP addresses to the workspace IP access list
1. Follow the steps outlined here to add the IP addresses for the NAT gateways collected above to your workspace IP ACL policy:
   https://docs.databricks.com/en/security/network/front-end/ip-access-list-workspace.html
3. Test that your deployment was successful 
1. Log in to your workspace
2. Navigate to "Preview" > "View All" 
3. Find "Enforce IP access list on Compute Plane Requests". On toggle on, IP ACL will be enforced on your NAT IP
4. Wait for up to 10 minutes for the config to be applied to the workspace.
5. Create and run a python notebook with a new cluster of any type except serverless.
   

   Cell #1 

   %pip install databricks-sdk --upgrade
   dbutils.library.restartPython()
   

   Cell #2

   from databricks.sdk import WorkspaceClient
   
   w = WorkspaceClient()
   w.clusters.list()
   

   If the code sample works, then your IP access list is set up correctly.
6. In case of failures, toggle off "Enforce IP access list on Compute Plane Requests". Wait for up to 10 minutes for the config to be applied to the workspace.

[SathwickKollipa]

We are not using NAT gateway and using Enterprise proxy for the compute resources. What is the solution for our case?

[AlexEsibov]

@SathwickKollipa  thanks for the question. You should allow the public IPs that are used for outbound traffic for your Databricks classic compute plane. If this goes through a proxy that obfuscates the IPs, then it makes sense to use the public IPs of that proxy.

[SathwickKollipa]

@AlexEsibov , we don't allow any outbound traffic from our VPC/subnets. By default, any cluster's we spin up don't have ability to talk to internet. If it needs internet connectivity, we set proxy thru init script of the cluster. 

In route table, the traffic for internet is routed to TGW and from their it routes to proxy. How does it works in our case?

[krikotti]

we don't use the NAT Gateway IP at our VPC level which using customer managed VPC for the Databricks. we use the IP whitelisting in our environment , do we still needs to whitelist anything? or not required? 

[sactom]

Hi @AlexEsibov  Does this apply to Premium Tier?

As in the fix it says need to be in Enterprise Pricing tier.

[AlexEsibov]

@SathwickKollipa if I understood your set-up correctly, all compute plane traffic will egress through proxy before hitting the control plane. In this case, allow-listing the IPs of the proxy in the workspace IP ACL should be sufficient. 

@krikotti can you clarify what you mean by "IP whitelisting in our environment"?

@sactom the workspace IP ACL list is available on the enterprise license tier, so this comm is not applicable to customers using the premium tier.