The behavior you are observing is expected for Serverless Compute created during or converted from an Express Setup trial and upgrading to Premium pay-as-you-go does not automatically remove these serverless egress restrictions.
Root Cause
When a Databricks workspace is created via Express Setup (or during the initial trial period), Databricks provisions Serverless Compute inside a Databricks-managed AWS VPC protected by a restrictive Default Serverless Egress Policy.
Default Restricted Allowlist: By default, Express Setup serverless environments restrict egress traffic via a managed DNS firewall to a tight allowlist of standard developer infrastructure domains (e.g., PyPI, GitHub, AWS S3, Databricks control plane).
DNS Refusal (rcode 5): Unapproved destinations (like news.mit.edu or [www.coindesk.com](https://www.coindesk.com)) are dropped directly at the Databricks-managed DNS firewall layer, producing the access_type = DROP and DNS rcode 5 (REFUSED) entries in system.access.outbound_network.
Account Console Access: Custom Account-Level Network Egress Policies (which allow configuring custom allowlists or switching to unrestricted internet egress for serverless) require the Enterprise tier. Since your workspace is on the Premium tier, the Egress Control UI in the Account Console is unavailable.
Resolution Options
Option 1: Use Classic Compute (Immediate Fix)
The serverless network restrictions apply only to Serverless Compute managed inside Databricks VPCs.
If you run your daily RSS and API fetching jobs on Classic Compute (single-node or multi-node clusters running in your own AWS account VPC via Customer-Managed VPC setup, or standard non-serverless job clusters):
Create a standard Job Cluster or All-Purpose Cluster in your workspace.
Route your daily ingestion notebooks to execute on this classic compute cluster.
Outbound HTTP/DNS calls will bypass the Databricks Serverless DNS firewall and use your AWS VPC's standard Internet Gateway / NAT Gateway setup, granting unrestricted access to public news and financial APIs.
Option 2: Request Manual Policy Relaxation from Databricks Support / Account Team
Although self-service account-level egress policy management requires Enterprise tier, Databricks internal operations can manually remove or relax the default express serverless egress restriction on Premium trial/converted workspaces upon request.
Since you do not have a paid Support Plan to open a formal ticket via the Support Portal:
Post on the Databricks Community Forums:
Create a post under the Workspace Management & Infrastructure or Databricks on AWS board in the Databricks Community Forum.
Include your Workspace ID (from your URL https://adb-.azuredatabricks.net), region, and a description of your issue:
"Upgraded workspace from Express trial to Premium pay-as-you-go. Serverless compute egress is still blocked by default trial DNS firewall (access_type = DROP, rcode 5). Need default serverless egress policy lifted for arbitrary public API ingestion."
Databricks Community Managers and Developer Advocates regularly monitor these posts and can escalate your Workspace ID internally to engineering to unblock egress.