topic Re: Serverless egress - public internet access issues in Data Engineering

Serverless egress - public internet access issues

MPD2 — Thu, 16 Jul 2026 09:17:04 GMT

I've just setup a new Databricks account on AWS with a workspace with "Serverless compute and default storage" and I am unable to access the public internet from notebooks and jobs for all but a handful of domains.

A subset works fine, I've identified it can connect to pypi.org, github.com, docker.com, databricks.com

response = requests.get("https://www.databricks.com/robots.txt") display({"status_code": response.status_code, "ok": response.ok}) Returns... {'status_code': 200, 'ok': True}

But others fail...

response = requests.get("https://www.google.com/robots.txt") display({"status_code": response.status_code, "ok": response.ok}) Responds... (": Failed to resolve 'www.google.com' ([Errno -3] Temporary failure in name resolution)"))

What I've verified so far...

Network policy: Serverless egress control is on the default policy, set to Full access (unrestricted outbound).
NCC: No Network Connectivity Configuration is attached to the workspace.
There are no entries in the system.access.outbound_network table.

Bypassing DNS entirely does not help, connecting directly to a resolved IP address fails with "Network is unreachable" so this isn't a resolver problem, I think it's a routing/egress problem at the network layer.

I have tried defining a custom Network policy in the account testing both Serverless egress control with "Allow access to all destinations" and "Restricted access to specific destinations" with a subset of domains but that has not had any impact (I have been terminating and restarting the serverless compute between each change and executing the notebook).

I have also tried setting up new workspaces in different AWS regions but the same problem persists.

My understanding of Serverless compute is that it is running within a Databricks managed network environment and not my AWS account is that correct? i.e. it's not running within a VPC where I have control over ingress/egress

Any help or suggestions appreciated.

Re: Serverless egress - public internet access issues

GabFernandes — Thu, 16 Jul 2026 12:42:17 GMT

Great debugging so far! A few things to check:

Yes, serverless runs in Databricks-managed infrastructure, not your AWS account or VPC — so you have no direct control over the underlying network routing. This is by design.

The domains that work for you (pypi.org, github.com, docker.com, databricks.com) are Databricks system-level allowlisted domains that are hardcoded at the infrastructure layer for platform operations (package installs, runtime setup, etc.) — regardless of your network policy settings. The fact that only those work is actually the fingerprint of a restricted policy being in effect, not "Full access."

The most likely culprit: your custom network policy was created at the account level but never attached to the workspace. Creating it doesn't auto-apply it. You need to explicitly associate the policy with the workspace in the account console (Account Console → Workspaces → select your workspace → Network Policy). Check this for both the "Full access" and the custom policies you tried.

One more gotcha from the docs: changing the internet access mode requires an explicit compute restart — just terminating the serverless cluster between changes should be enough, but confirm the policy attachment is showing on the workspace detail before restarting.

If the policy IS attached and you're still blocked, this is worth opening a support ticket — new AWS accounts with the simplified "Serverless compute and default storage" setup occasionally have account-level provisioning issues that support needs to fix on the backend.

As an immediate workaround, Classic compute running inside your own AWS VPC will give you full egress control.

Re: Serverless egress - public internet access issues

MPD2 — Thu, 16 Jul 2026 13:49:03 GMT

Thanks for the suggestions @GabFernandes
I double checked that I had explicitly attached the policy to my workspace.

One amendment to my original post, I am seeing entries in the system.access.outbound_network table with lots of entries with access_type DROP.

Am I correct in thinking I can only open a support ticket if I have a support contract with Databricks? This is only for a personal project so I don't.

Re: Serverless egress - public internet access issues

GabFernandes — Thu, 16 Jul 2026 14:01:05 GMT

Hey, thanks for confirming that and for the extra detail!

You're right — official support tickets with SLA require an active Databricks support contract, which typically isn't available for personal/free-tier usage (Free Trial credits or Free Edition). For projects like this, the best path is sticking with the Community here, or the Databricks Community Slack — plenty of product folks hang around there too.

That said, since you're already seeing DROP entries in system.access.outbound_network, you can actually self-diagnose without needing a ticket. Try drilling into the exact blocked destination and reason:

SELECT destination_type, destination, dns_event, storage_event.rejection_reason, event_time
FROM system.access.outbound_network
WHERE access_type = 'DROP'
ORDER BY event_time DESC
LIMIT 20;

This usually shows whether the policy is missing a rule for that specific domain/IP/storage path — which is a much more common cause than the policy simply not being attached. If the destination looks like it should be allowed and still shows DROP, that's worth flagging as a bug report on the Community rather than a support ticket, since it'd help others too.

Re: Serverless egress - public internet access issues

MPD2 — Thu, 16 Jul 2026 14:49:30 GMT

They're all showing this...

DNS www.google.com {"domain_name":"www.google.com","rcode":5,"rdata":[]} null 2026-07-16T13:22:02.000+00:00

The rejection reasons are all `null`. The policy I am using has egress configured as "Allow access to all destinations" so I would not be expecting www.google.com to be dropped. I'm assuming this indicates it is probably a bug.

You mentioned...

new AWS accounts with the simplified "Serverless compute and default storage" setup occasionally have account-level provisioning issues that support needs to fix on the backend.

I'm guessing without a support contract I'm a bit stuck with this if it is a bug 😅

Re: Serverless egress - public internet access issues

binlogreader — Sat, 18 Jul 2026 19:53:43 GMT

Hi there, from your note and comments, I couldn't gather if you have tried standard compute. Did you experience the same issues with standard compute? With serverless, you might experience some snags since the infrastructure is owned by Databricks and you will need to work their infra team to unpack some of the roadblocks.

Re: Serverless egress - public internet access issues

MPD2 — Sat, 18 Jul 2026 22:03:46 GMT

I hadn't tried standard compute at the time, I have now and internet access works for all domains from the EC2 instances. So I'm using a mix of Serverless for workloads where I don't need to access the internet and standard for the places I do.