@el_mark
Root Cause
Serverless SQL or notebook queries that whitelist the Databricks serverless cluster IP succeed because those IP addresses are allowed access through the Azure storage account firewall
Dedicated compute instances or non-serverless SQL warehouses on AWS typically use different IP addresses (e.g., from AWS VPC egress) that must be explicitly allowed on the Azure storage account firewall.
Solution:
Storage Firewall Rules
Verify the Azure Storage Account firewall includes the egress IP addresses used by your dedicated AWS VPC and non-serverless compute instances.
Network Connectivity Configuration
If your dedicated compute uses private IPs or VPC security groups, ensure proper routing and DNS resolution to access Azure storage endpoint URLs internally
Delta Sharing IP Access List
If using Delta Sharing IP access lists on the provider side, add the dedicated IP addresses to those lists to ensure data access.
Cross-Cloud Access Best Practices
Check that the IPs or CIDR blocks your dedicated AWS VPC compute uses are explicitly allowed in Azure Storage firewalls.
Azure Databricks Delta Sharing troubleshooting guide: https://learn.microsoft.com/en-us/azure/databricks/delta-sharing/troubleshooting
IP restrictions and access controls for Delta Sharing: https://learn.microsoft.com/en-us/azure/databricks/delta-sharing/access-list
