Serverless DNS failures after trial upgrade (AWS us-east-2)

[revian]

Our AWS Premium serverless workspace in us-east-2 cannot resolve several public domains. We added a payment method, and the account UI confirms that the trial upgrade is complete.

Three fresh serverless notebook runs on September 24, 2026, at approximately 15:01, 15:04 and 15:13 UTC produced the same results using Python's socket.getaddrinfo:

• app.bellesoftware.com.br, www.google.com, and example.com: gaierror: [Errno -3] Temporary failure in name resolution.
• pypi.org and www.databricks.com: resolve successfully.

An HTTPS request to the external API fails with requests.exceptions.ConnectionError before API authentication. That API is reachable from our local computer.

The account network-policy API reports default-policy as FULL_ACCESS with ENFORCED mode and no blocked destinations. However, this is a Premium workspace, and the documentation says default policies apply only to Enterprise workspaces, so we do not assume that response establishes the effective egress permissions.

Could a trial-related DNS or outbound restriction remain after the upgrade? How can we verify whether one is still applied and have it removed? If this is unrelated to trial restrictions, which serverless networking configuration or diagnostic should we check next?

This is blocking the initial load for a daily REST API ingestion job. We can provide workspace details and diagnostic run IDs privately to Databricks support.

[revian]

My account Id: e5beacb1-d0a7-44cd-8de6-8d6f89d5af81

[ivanvyd]

@revian thanks for the detailed diagnostics. If the account used express setup, Databricks notes that some trial restrictions can persist after upgrading. A remaining network restriction is worth checking, but isn't confirmed here.

If available, check system.access.outbound_network for DNS events matching your test timestamps. A matching DROP indicates an enforced denial; DRY_RUN_DENIAL doesn't. No matching events would be inconclusive, since coverage of trial restrictions on Premium isn't clearly documented.

[lmcorreahdb]

One additional diagnostic could help distinguish a workspace-level trial restriction from a standard serverless egress policy.

Since serverless network policies currently require Enterprise tier, and the default policy is only applied to Enterprise workspaces, the FULL_ACCESS response from the account API would not necessarily explain the effective behavior of this Premium workspace.

I would run the same DNS test while querying system.access.outbound_network for the exact workspace_id, timestamps, and failing destinations.

If there is a DROP event, that gives Databricks Support a concrete policy enforcement event to trace.

If there is no event, I would also test the same domains from a newly created post-upgrade workspace in the same region, if your account allows it. If the new workspace resolves them while the original Express Setup workspace does not, that would be a useful signal that the restriction is tied to the original trial workspace rather than the destination itself.

Given that Databricks documentation notes that some Express Setup trial limitations may remain after upgrading, I would include both workspace IDs and the failing run timestamps in the support case and ask them specifically to verify whether any internal trial-era serverless egress restriction is still attached to the original workspace.

I would also avoid treating the account default-policy response as proof of effective egress configuration for this Premium workspace.

[revian]

Thanks, the denial-log suggestion helped. We found historical and fresh DNS events with access_type=DROP, rcode=5, and network_source_type=General Compute.

Three new serverless runs reproduced the selective DNS failures, while the same domains resolved locally. The workspace is confirmed as Premium, and the account default policy still reports FULL_ACCESS.

This confirms enforced denial, but not whether it comes from a retained trial restriction. How can we identify the responsible control and have it removed? I can provide event and run IDs privately to Databricks Support.

[dbxenthusiast]

@revian We are also seeing a similar issue where serverless can't access the public internet although egress is enabled. We are also on Premium and using AWS as our cloud.
Another issue we are facing is that we are unable to access any Unity Catalog external location (S3) using serverless; we are getting 403 for this. Is that also something happening for you?

[lmcorreahdb]

Interesting, the public internet issue does sound very similar to the original case.

For the S3 403, though, I would troubleshoot it separately first because it could be a Unity Catalog / AWS authorization issue rather than the same outbound restriction.

A few checks that could help isolate it:

1. In Catalog Explorer, validate the Storage Credential configuration and run “Test connection” on the External Location.
2. Confirm that the External Location and its Storage Credential are available to the current workspace if workspace bindings are being used.
3. Check the S3 bucket policy for conditions based on source VPC endpoints, source IPs, or older Databricks networking assumptions. Serverless uses Databricks-managed networking, so policies that work for classic compute can still return 403 from serverless.
4. If the S3 bucket is in the same region, Databricks recommends allowing serverless access using the published aws:VpceOrgPaths condition rather than relying on classic-compute network identifiers.
5. If possible, compare the same External Location from classic compute and serverless. If classic works and serverless returns 403 using the same Unity Catalog object, that would strongly narrow the issue to the networking/bucket-policy path rather than the storage credential itself.

If you can share whether the S3 bucket is in the same AWS region as the workspace/metastore, and whether the External Location test succeeds, that would help narrow it down further

[dbxenthusiast]

1. Test Connection and Validate Credentials both work fine.
2. External location and storage credentials are fine.
3 & 4. Bucket policy is clean, no Deny, no VPC endpoint, no Source IP conditions
5. Classic compute is working fine for us. On doing LIST '<s3 bucket>' we are getting Access to storage location denied because of serverless network policy with Request Id: Null (so request never reached s3)
We are on premium pay as you go plan and can't see the option to change egress rules, so it should be on by default. I have another account on Enterprise and the default egress was allow access to all storage location, there as well we have the same issue.
Another thing to note is public internet also cannot be accessed using serverless in notebooks, like trying to hit google url also throws a connection error.
Would appreciate if someone from Databricks could help.

[lmcorreahdb]

Thanks for sharing the additional tests @dbxenthusiast. These results narrow the problem down quite a bit

Since:

Storage Credential validation succeeds,
the External Location works from classic compute,
the S3 bucket policy has no restrictive VPC/IP conditions,
serverless returns an explicit “denied because of serverless network policy” message,
and the request does not appear to reach S3,
I would no longer focus primarily on IAM or the bucket policy.

The next useful step would be to correlate the failing S3 test with system.access.outbound_network and check whether you also get a STORAGE / DROP event, similar to the DNS DROP events seen in the original case.

It would also be interesting to compare the Premium workspace and the Enterprise workspace separately:

For the Enterprise workspace, I would verify not only the default policy, but also whether an explicit network policy is assigned directly to that workspace, since a workspace-specific assignment could explain why “Full Access” on the default policy does not match the observed behavior.

For the Premium workspace, the behavior seems even more interesting because current Databricks documentation states that serverless network policies require Enterprise tier. If a Premium workspace is being blocked by an effective serverless network policy, that seems worth escalating to Databricks Support with the workspace ID, outbound_network event IDs, timestamps, and the “Request Id: Null” error.

Since both public DNS and Unity Catalog storage access are failing from serverless while classic compute works, this now looks more like a common serverless egress enforcement issue than two independent S3/DNS problems.

If you can share whether the S3 failure also appears as a STORAGE / DROP event in system.access.outbound_network, that would be a useful next data point.

[anuj_lathi]

Short version: rcode=5 (REFUSED) with access_type=DROP means a serverless egress policy is actively refusing those lookups. That is a policy-enforcement signature, not a flaky resolver. I can't see your account, so I can't rule out a leftover trial restriction. But the pattern points to a network policy being applied to this workspace, and I'd chase that before assuming a trial flag.

Why this looks like policy, not DNS flakiness

• Serverless egress control filters at the DNS layer, so a blocked FQDN fails to resolve. That's why requests dies with a ConnectionError before it ever reaches auth.
• Some domains resolve (pypi.org, www.databricks.com) while others are refused (google.com, example.com, your .com.br API). That is what an allowlist does: a small set of destinations is permitted and everything else is dropped. A random network fault wouldn't be this consistent across runs.
• Your own logs show the same thing: DROP events with rcode=5, repeatable, and local resolution works fine.

Don't treat default-policy as the effective policy

The default policy is only what applies to workspaces with no explicit assignment. A workspace can be bound to a different policy, and that binding is what matters. So default-policy = FULL_ACCESS tells you nothing about this workspace if it's attached to another one.

Check these, in order:

1. Which policy is this workspace actually bound to? Look at the workspace's network policy assignment in the account console (Security → Network policies), or via the account-level workspace network option / workspace GET API, which returns the assigned network_policy_id. Don't assume it equals default-policy.
2. Fetch that specific policy by ID and look at its egress settings: restriction mode (FULL_ACCESS vs RESTRICTED_ACCESS), allowed internet destinations (FQDNs), and allowed storage destinations. If it's restricted and your API's domain isn't listed, that's your answer.
3. Check for dry-run/enforcement mode on the policy. Your logs show real DROPs, so enforcement is happening, but confirm which policy generated them.
4. Correlate the denial events with the policy. Query the outbound network denial events in the system tables for your workspace ID and the destinations in question. The destination, access_type, and timestamps let you line up each drop against a policy change or assignment time. Pay attention to when the earliest DROP events start. If they predate your upgrade and continue unchanged after it, that's the signal that something was never re-evaluated.

Fixing it

If the workspace is bound to a restricted policy:

• Add the needed FQDNs (e.g. your API host) to the allowed internet destinations, or
• Reassign the workspace to a policy with full internet access.

Then start a new serverless session and retest with socket.getaddrinfo. Policy changes take a few minutes to propagate, and existing sessions may hold the old config.

If the policy is genuinely FULL_ACCESS and the workspace is bound to it

Then the control is not something you can see or edit in the account-level network policy API, and that is exactly when you need Databricks to look at the backend. Whether a trial-era restriction can linger after an upgrade isn't something I can confirm from the outside, and it isn't something you can inspect or remove yourself. Open a support ticket (the AWS/Premium entitlement is yours to cite) and include:

• Workspace ID and region (us-east-2)
• The DROP / rcode=5 event IDs and timestamps, including the earliest ones
• The serverless run IDs from September 24
• The network policy ID the workspace is bound to and its full JSON
• The date you upgraded, to show the drops continue after it
• A direct question: "What control is generating these DNS refusals for network_source_type=General Compute, and is it tied to the workspace's trial/upgrade state?"

Ask them to confirm the workspace's effective serverless egress configuration, since that is what you can't see from your side.

Unblocking the daily job in the meantime

If the ingestion can't wait for support, run that one job on classic compute in your own VPC. Egress there goes through your VPC's NAT and DNS, not the serverless policy. Then move it back to serverless once the egress control is sorted.

[dbxenthusiast]

Thanks for the detailed replies.

Network policy is not visible or editable (Enterprise only per docs). There is only one network policy with the name default-policy (which i can't open in my premium plan) no other network policies are present.

LIST succeeds on the bucket used as catalog managed storage, but every object level read fails: HEAD on delta log for UC managed tables and read_files on a plain CSV.

outbound_network table has no rows for these denials; so unlike the DNS case there is no DROP event to trace.

Account started as a free trial on our own AWS account (not Express Setup), subscribed via AWS marketplace.

I've opened a support case 01040556 with full details. @anuj_lathi if you're able to flag it internally or check whether a trial-era serverless egress restriction can produce storage denials that don't appear in outbound_network, I'd really appreciate it.

[dbxenthusiast]

@anuj_lathi do you think the reason behind this issue could be similar to https://community.databricks.com/t5/generative-ai/premium-payg-account-stuck-at-trial-verified-rate-... ?
Can trust tier of TRIAL_VERIFIED impact the serverless egress to be disabled by databricks itself?
We are on Pay as you go plan through AWS marketplace and trial has ended but we are yet to make our first payment.