Unauthorized network access to workspace

[nsingh_tl]

We're setting up a new workspace on AWS commercial cloud (customer-managed VPC, back-end PrivateLink) and are blocked from using Unity Catalog due to an account-level network policy restriction.

Issue:

Notebook calls to Unity Catalog fail with:

Error Code: UNEXPECTED_HTTP_ERROR

Message: HTTP request failed with status: HTTP/1.1 403 Forbidden, original HTTP response body: Unauthorized network access to workspace

[balajij8]

@nsingh_tl 

Unauthorized network access to workspace 403 error is generally a network layer block, not an IAM or permissions issue. When you are running a customer-managed VPC with back end PrivateLink, Unity Catalog needs to connect to the Databricks regional hostname rather than the specific workspace URL. The underlying issue here is likely that the VPC endpoint's Private DNS names option isn't enabled on the AWS side. When that is disabled, the regional hostname resolves to a public IP instead of routing through the private VPC endpoint. If the VPC is fully private and lacks a NAT gateway, that connection drops and throws 403 error. To isolate and fix this, you can check the AWS Console under VPC - Endpoints, open the Databricks workspace VPC endpoint, and verify that Private DNS names enabled is set to Yes. VPC must have both enableDnsSupport and enableDnsHostnames set to true.

If its already configured, confirm the account-level network policy in Databricks. If it is set to Restricted Access mode, it governs serverless compute egress and denies all outbound traffic by default except for Unity Catalog external locations and explicitly allow listed FQDNs or S3 buckets. Ask the account admin to check it in the Security - Networking - Context-based ingress & egress control. You can switch the policy to Full Access or manually add the required destinations in the allowed list.

Ensure that the Unity Catalog metastore is attached to this specific workspace (Account Console - Workspaces - check the Metastore). Ensure the workspace has the UC mapped and attached.

The pipelines rely on Unity Catalog for governance, and you won't be able to ingest your Oracle data or build out that silver layer until this is resolved. You can check the AWS-side VPC endpoint Private DNS, followed by the Databricks-side network policy and metastore attachment. If the 403 persists after fixing all sides, you can check with databricks support.

[nsingh_tl]

Thank you @balajij8 for your input. The metastore is attached to the workspace, I am able to create schemas from the control plane ( workspace), but when I use notebook and compute to create a table or schema, I am getting "Unauthorized network access to workspace". This is a new account created using AWS Marketplace, I do not have support contract with Databricks and no available from them,  
VPC endpoint have private DNS enable.

Relay is point to: tunnel.privatelink.cloud.databricks.com (com.amazonaws.vpce.us-west-1.vpce-svc-04cb91f9372b792fe)
Dataplane is point to: ncalifornia.privatelink.cloud.databricks.com (com.amazonaws.vpce.us-west-1.vpce-svc-09bb6ca26208063f2)

[balajij8]

@nsingh_tl 

Your back end PrivateLink only serves classic compute - clusters running inside your customer-managed VPC. Serverless compute generally runs in Databricks managed infrastructure outside VPC and reaches the workspace control plane API via public IP addresses, bypassing the Private Link. This is an ingress rejection (the workspace actively blocks the connection).

You can add the serverless compute outbound IPs to the workspace's allowed list. More details here. Download it and filter for entries where service = "Databricks", type = "outbound", platform = "aws", and region = "us-west-1". Add the resulting CIDR blocks to the workspace's IP access list (Admin Settings - IP Access Lists) or to the context-based ingress rules in account-level network policy (Account Console - Security - Networking - Ingress tab). You can setup periodic refresh on the allowlist regularly.

You can alternatively use classic compute - create a cluster that deploys into the customer-managed VPC subnets. Classic compute uses existing back end Private Link to reach the control plane privately and generally never hits the IP access list.

[nsingh_tl]

Serverless compute is disabled on my account for some reason, and it won't let me turn it on. 

My compute is getting provisioned in the same VPC, private subnet, and SG is also associated with VPC as well.

I am not able to this option to add IP CIDR, `Add the resulting CIDR blocks to the workspace's IP access list`, only IP access list is restrict the IP from workspace acces, currently that option is disabled. 

[ivanvyd]

@nsingh_tl given the latest useful details, I'd check the classic-compute PrivateLink path first.

From the affected cluster, run %sh nslookup <workspace-url> and confirm that the workspace hostname resolves to the private IP for the registered workspace REST API endpoint. Also check the workspace network configuration has both required VPC endpoints registered: the SCC relay and workspace REST API endpoints.

If DNS already resolves privately, check Private Access Settings next. With access level ENDPOINT, the relevant REST API VPC endpoint must be allowed.

If context-based ingress is also configured, it has to allow the same endpoint. system.access.inbound_network can help confirm an ingress-policy denial.

That should narrow this down before looking at UC permissions. Please, let us know the resukts.

[nsingh_tl]

Here is output: 

%sh

nslookup dbc-xxxxxxxxx-xxxx.cloud.databricks.com

Output: 

Server: 10.10.0.2
Address: 10.10.0.2#53
Non-authoritative answer:
dbc-xxxxxxxxx-xxxx.cloud.databricks.com canonical name = ncalifornia.privatelink.cloud.databricks.com.
Name: ncalifornia.privatelink.cloud.databricks.com
Address: 10.10.x.x
Name: ncalifornia.privatelink.cloud.databricks.com
Address: 10.10.x.y

%sql

`select * from system.access.inbound_network` give below error:
"[UC_UNCAUGHT_CLIENT_EXCEPTION] Encountered an unexpected HTTP error while communicating with the Databricks Unity Catalog backend. Error Code: UNEXPECTED_HTTP_ERROR. Message: HTTP request failed with status: HTTP/1.1 403 Forbidden, original HTTP response body: Unauthorized network access to workspace:"

My settings:

Private Access Level

Specifies which VPC endpoints can connect to the attached workspace

Account - Limit connections to those VPC endpoints that are registered in your Databricks account.

[ivanvyd]

@nsingh_tl thanks, that confirms private DNS resolution for the workspace hostname.

1) Could you compare both returned addresses with the network-interface IPs of your workspace REST API VPC endpoint in AWS? They should match. Private addresses alone do not confirm that the intended endpoint is being used.

With Private Access Level = Account, the endpoint must be registered in your Databricks account, not just exist in your AWS account. You do not need an explicit ENDPOINT allowlist with that setting.

In the Databricks account console (like Account Console > Security > Networking > VPC endpoints) , check that the actual workspace and SCC endpoint IDs (vpce-...) are registered in us-west-1. The vpce-svc-... values you shared identify the services, not your individual endpoints. Also check that the workspace’s attached network configuration references those registrations in the correct workspace and SCC fields. The endpoint registration documentation covers this distinction.

2) Next, try a workspace REST API request from the affected cluster. This uses the same API as Databricks' documented connectivity test . Run it in a Python cell, replacing the hostname and existing secret scope/key. Use a PAT already authorized to list tokens, rather than granting additional permissions just for this test.

import requests

token = dbutils.secrets.get(
scope="<secret-scope>",
key="<pat-key>",
)

response = requests.get(
"https://<workspace-hostname>/api/2.0/token/list",
headers={"Authorization": f"Bearer {token}"},
timeout=20,
)

print("HTTP", response.status_code)

if response.status_code >= 400:
print(response.text)

• The same 403 "Unauthorized network access to workspace": the rejection also affects a non-UC workspace API. I'd continue checking endpoint registration and workspace network controls, including any attached context-based ingress policy.
• HTTP 200: this particular request works. That narrows the investigation but does not prove that every UC request is permitted.
• An authentication or scope error: resolve that before using this test to draw conclusions about the original network restriction.

The system.access.inbound_network query also depends on Unity Catalog, since system tables are UC-governed. My earlier suggestion therefore cannot provide the denial logs from this cluster while UC access is failing.

Please share only the HTTP status and redacted error, not the token or successful token-list output.

[nsingh_tl]

Here is the output of the screenshot. The response is 200 for a non-UC call from compute. One more thing: this workspace and account are set up for a HIPAA-compliant workload. 

[ivanvyd]

@nsingh_tl the 200 is for the token API. I'd compare a UC REST request with notebook SQL before changing any network settings.

From the same cluster, run this read-only List catalogs request. Reuse your existing token variable, provided the token belongs to the user running the notebook. If the token is scoped, it must allow the unity-catalog API scope.

import requests

response = requests.get(
    "https://<workspace-hostname>/api/2.1/unity-catalog/catalogs",
    headers={"Authorization": f"Bearer {token}"},
    params={"max_results": 1},
    timeout=20,
    allow_redirects=False,
)

print("HTTP", response.status_code)

if response.status_code >= 400:
    print(response.text[:1000])

Then run SHOW CATALOGS in a SQL cell attached to that same cluster:

SHOW CATALOGS;

If REST returns 200 but SQL returns the original network-access 403, check Compute > your cluster > Driver logs around the SQL failure. Please include the runtime version, access mode and redacted exception, with any destination hostname, port or request ID that appears. The driver-log documentation explains where to find those logs.

If REST also fails, please share its http status and redacted error. A token-scope or privilege error is different from the original "Unauthorized network access to workspace" response.

The results so far do not establish HIPAA settings, SRA or a blocked port as the cause.

[nsingh_tl]

I am getting return code of 200 from api call  `show the catalog api`, but show catalogs failed again

[ivanvyd]

@nsingh_tl these results support the regional-DNS explanation @data_pulse raised. Your logs show public DNS resolution, not a missing route-table entry.

Databricks documents UC initialization contacting a regional hostname directly, bypassing the workspace URL’s PrivateLink CNAME chain. If the failing UC request uses that hostname, this could explain the difference between your REST and SQL results.

Since you already reported private DNS enabled, I’d check the actual workspace REST API endpoint’s DNS configuration, rather than repeat the workspace lookup. Run this read-only AWS CLI command from an authorized AWS terminal, replacing the endpoint ID:

aws ec2 describe-vpc-endpoints \
  --region us-west-1 \
  --vpc-endpoint-ids "<workspace-rest-vpce-id>" \
  --query 'VpcEndpoints[0].{PrivateDNS:PrivateDnsEnabled,DNSNames:DnsEntries[].DnsName}' \
  --output json

If PrivateDNS is false, enable it on that workspace endpoint through VPC > Endpoints > Actions > Modify private DNS name. Both VPC DNS resolution and DNS hostnames must be enabled. AWS documents these requirements here.

If it is already true, check whether the private DNS configuration covers the hostname UC actually uses, and inspect Route 53 Resolver rules associated with the VPC. A forwarding rule for the same domain can take precedence over a private hosted zone, so an enabled endpoint setting alone does not establish which DNS answer the cluster receives. AWS explains that precedence here.

[ivanvyd]

Before creating a manual DNS override, confirm the destination hostname in the failing UC request. The current regional reference lists oregon.cloud.databricks.com for us-west-1.

The correction to test is private resolution of that actual UC hostname to the intended workspace endpoint, followed by another SHOW CATALOGS attempt. I would leave the ingress restrictions unchanged while checking this.

[nsingh_tl]

Here is output; private DNS is enabled. 

~ $ aws  ec2 describe-vpc-endpoints \
>   --region us-west-1 \
>   --vpc-endpoint-ids "vpce-005a4903xxxxxxx" \
>   --query 'VpcEndpoints[0].{PrivateDNS:PrivateDnsEnabled,DNSNames:DnsEntries[].DnsName}' \
~ $ aws  ec2 describe-vpc-endpoints \
>   --region us-west-1 \
>   --vpc-endpoint-ids "vpce-005a4903xxxxxxx" \
>   --query 'VpcEndpoints[0].{PrivateDNS:PrivateDnsEnabled,DNSNames:DnsEntries[].DnsName}' \
>   --output json
{
    "PrivateDNS": true,
    "DNSNames": [
        "vpce-005a4903xxxxxxx-k5bwn2b7.vpce-svc-09bb6ca26208063f2.us-west-1.vpce.amazonaws.com",
        "vpce-005a4903xxxxxxx-k5bwn2b7-us-west-1a.vpce-svc-09bb6ca26208063f2.us-west-1.vpce.amazonaws.com",
        "vpce-005a4903xxxxxxx-k5bwn2b7-us-west-1c.vpce-svc-09bb6ca26208063f2.us-west-1.vpce.amazonaws.com",
        "ncalifornia.privatelink.cloud.databricks.com"
    ]
}
~ $ 

[ivanvyd · Accepted solution]

@nsingh_tl good, I would leave that setting enabled.

AWS private DNS maps the endpoint service’s configured DNS name. This output does not establish which destination the failing SQL request reaches, so it does not yet confirm or rule out the regional-DNS explanation.

Could you reproduce SHOW CATALOGS, note the failure timestamp, and check Compute > your cluster > Driver logs, particularly Log4j and stderr? Databricks documents those logs here.

Please share the Runtime version, access mode and a redacted exception excerpt, including its nested causes. Include any hostname, port or request ID if present; those details may not be logged.

I would hold off on manual DNS overrides or changes to ingress restrictions until we have evidence from the failing request. Please remove credentials and sensitive workload data from the excerpt.