I've just setup a new Databricks account on AWS with a workspace with "Serverless compute and default storage" and I am unable to access the public internet from notebooks and jobs for all but a handful of domains.

A subset works fine, I've identified it can connect to pypi.org, github.com, docker.com, databricks.com

response = requests.get("https://www.databricks.com/robots.txt")
display({"status_code": response.status_code, "ok": response.ok})

Returns...
{'status_code': 200, 'ok': True}

But others fail...

response = requests.get("https://www.google.com/robots.txt")
display({"status_code": response.status_code, "ok": response.ok})

Responds...
(": Failed to resolve 'www.google.com' ([Errno -3] Temporary failure in name resolution)"))

What I've verified so far...

Network policy: Serverless egress control is on the default policy, set to Full access (unrestricted outbound).
NCC: No Network Connectivity Configuration is attached to the workspace.
There are no entries in the system.access.outbound_network table.

Bypassing DNS entirely does not help, connecting directly to a resolved IP address fails with "Network is unreachable" so this isn't a resolver problem, I think it's a routing/egress problem at the network layer.

I have tried defining a custom Network policy in the account testing both Serverless egress control with "Allow access to all destinations" and "Restricted access to specific destinations" with a subset of domains but that has not had any impact (I have been terminating and restarting the serverless compute between each change and executing the notebook).

I have also tried setting up new workspaces in different AWS regions but the same problem persists.

My understanding of Serverless compute is that it is running within a Databricks managed network environment and not my AWS account is that correct? i.e. it's not running within a VPC where I have control over ingress/egress

Any help or suggestions appreciated.