cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Serverless DNS requests to external API being DROPPED despite FULL_ACCESS network policy and no bloc

DemandDelphi
New Contributor II

Serverless DNS requests to external API being DROPPED despite FULL_ACCESS network policy and no blocked destinations

I'm troubleshooting outbound access from a Databricks production workspace to the Toast API at ws-api.toasttab.com.

Requests from Databricks fail during DNS resolution with the following errors:

urllib.error.URLError: <urlopen error [Errno -3] Temporary failure in name resolution>

socket.gaierror: [Errno -3] Temporary failure in name resolution

This initially looked like a normal DNS or network issue, but the Databricks system.access.outbound_network table showed approximately 30 DNS DROP events for ws-api.toasttab.com during the failed attempts.

That appears to establish that the DNS requests are being actively denied by Databricks outbound network enforcement rather than simply failing due to Toast or public DNS.

However, I cannot identify the configuration responsible for the denial.

The workspace is on the Premium tier.

The network policy returned for the account/workspace is FULL_ACCESS.

The raw policy response contains no blocked internet destinations or domains.

No customer-configured allowlist or denylist appears to explain the Toast restriction.

The destination is an ordinary external HTTPS API, not another Databricks workspace or a Private Link endpoint.

Access to Toast works successfully outside Databricks.

The workspace/account originally existed under a trial/free configuration before being upgraded, so I am wondering whether an older serverless egress restriction or another effective network control may still be attached somewhere that is not visible through the normal network policy configuration.

My main question is:

What Databricks configuration or internal enforcement layer can produce DNS DROP events in system.access.outbound_network when the visible/effective network policy is FULL_ACCESS and contains no blocked destinations?

If there is another API, account setting, workspace binding, or serverless network configuration I should inspect to identify the controlling rule, I would greatly appreciate guidance.

I can provide sanitized policy responses and outbound network event details if useful.

2 REPLIES 2

DemandDelphi
New Contributor II

It seems like this might just be blocked in Serverless... I'm building a custom Job in GCP to run the API call!

anuj_lathi
Databricks Employee
Databricks Employee

Short version: with an effective FULL_ACCESS policy, DNS drops shouldn't come from anything you can configure. I can't tell you which internal layer is responsible, but I can tell you how to confirm what's attached and how to get a definitive answer quickly.

What's going on

Serverless compute runs in the Databricks-managed serverless compute plane, not in your cloud account [1]. Egress from that plane is enforced by Databricks-side controls, and the account-level network policy is the only customer-facing control for it. If that policy really is FULL_ACCESS for this workspace, a DROP on DNS is unexpected. It points either to a different policy than the one you inspected, or to something on the platform side that you can't see.

What I'd check first

  1. Confirm which policy is bound to this workspace. Don't rely on the account default. A workspace can have its own network policy assigned, and that is what's enforced. Compare the policy ID on the workspace with the one you pulled. This is the most common mismatch.
  2. Check the workspace ID on the events. system.access.outbound_network is account-wide. Make sure the 30 DROP rows carry the production workspace ID and not another workspace that has a restricted policy.
  3. Check timing. Compare the event timestamps with when the policy was last changed. If the failures predate a switch to FULL_ACCESS, the drops may simply be stale.
  4. Read every column on the DROP rows. Look at the destination, the compute or workload type, and any field indicating the reason or the policy that matched. That should tell you whether it's a policy decision or something else.
  5. Run a control test from the same serverless compute. Resolve a known-good public host alongside Toast:
import socket
for host in ["ws-api.toasttab.com", "pypi.org", "example.com"]:
    try:
        print(host, socket.getaddrinfo(host, 443))
    except Exception as e:
        print(host, "FAILED:", e)

If everything fails, it's a general egress or DNS problem for that workspace. If only Toast fails, the restriction is destination-specific.
6. Compare compute types. Try a serverless notebook, a serverless job, and a classic cluster in the same workspace. If classic resolves Toast and serverless doesn't, the issue is isolated to serverless egress.

The trial-to-paid theory

It's plausible that something carried over from the trial. But if it exists, it isn't visible or editable through the network policy API or account console. You can't find it by inspecting configuration, so don't spend much time hunting for it.

Next step

If steps 1โ€“3 don't explain it, open a support ticket. They can see the effective enforcement state for your workspace, which you can't. Include:

  • the workspace ID and account ID
  • the policy ID and the raw policy JSON
  • a handful of DROP rows with timestamps and all columns
  • the result of the control test above

Ask specifically which enforcement rule matched those events.

Workaround

Your GCP job approach is reasonable and unblocks you right now. Classic compute in your own network gives you full control over DNS and egress, and it would also tell you whether the restriction is serverless-specific. Landing the Toast data in cloud storage or a Unity Catalog volume and reading it from Databricks is a clean pattern.

References

[1] Connect to serverless compute | Databricks on AWS โ€” https://docs.databricks.com/aws/en/compute/serverless

Anuj Lathi
Solutions Engineer @ Databricks