Hi @lubiarzm1,
This is a solid topic to discuss. This is an architecture challenge that many organizations encounter when working toward full private network isolation on Azure Databricks. Let me walk you through how to address the connectivity to the default workspace storage account (the dbstorageXXXXXXXXX account in your managed resource group).
UNDERSTANDING THE PROBLEM
When you create an Azure Databricks workspace, a storage account is automatically provisioned in the managed resource group. This storage account backs the DBFS root and also stores internal workspace system data (notebook revisions, job results, Spark logs, etc.). Your classic compute (Job Compute) needs to reach this storage account at abfss://root@dbstorageXXXXXXXXX.dfs.core.windows.net, and you are finding that you cannot easily add private endpoints or disable public access on it because it lives inside the Databricks-managed resource group.
The good news is: yes, full private network isolation IS achievable, including for this storage account. Here is how.
SOLUTION 1: ENABLE THE WORKSPACE STORAGE FIREWALL (RECOMMENDED)
Azure Databricks supports enabling a storage firewall on the workspace storage account (DBFS root). When you deploy or update your workspace with the correct ARM template parameters, Azure Databricks will configure the workspace storage account to deny public network access and set up the necessary service endpoints or private endpoints automatically.
Since you are using VNet injection, your classic compute resources can reach the workspace storage account via Azure service endpoints on your workspace subnets. Here is what to do:
1. Ensure your workspace uses VNet injection with Secure Cluster Connectivity (SCC / No Public IP) enabled -- it sounds like you already have this.
2. Add Azure Storage service endpoints to BOTH of your workspace subnets (the host/public subnet and the container/private subnet) in the Azure Portal:
- Go to your VNet, then Subnets
- Select each workspace subnet
- Under Service endpoints, add Microsoft.Storage
- Save
3. After adding service endpoints, the workspace storage account will be reachable from your VNet-injected clusters over the Microsoft backbone network, without requiring public internet access.
4. You can then restrict the workspace storage account's networking to deny public access. Navigate to the managed resource group, find the storage account (dbstorageXXXXXXXXX), go to Networking, and set Public network access to "Enabled from selected virtual networks and IP addresses." Add both of your workspace subnets as allowed networks.
IMPORTANT NOTE: The storage account in the managed resource group has restrictions on what you can modify. You can typically add network rules and service endpoints, but you should NOT modify other settings or delete any resources in the managed resource group. If you encounter permissions issues modifying the storage account's firewall settings directly, see the ARM template approach below.
SOLUTION 2: USE AN ARM TEMPLATE WITH STORAGE FIREWALL PARAMETERS
For a more robust approach, especially if you cannot modify the managed resource group storage account directly, you can configure the workspace storage firewall at deployment time using ARM template parameters. When deploying or updating your Azure Databricks workspace, include these parameters:
"storageAccountFirewall": {
"value": {
"enabled": true,
"defaultAction": "Deny"
}
}
Or, if using the Azure CLI to create/update the workspace, you can specify parameters to enable the storage account firewall. This tells Azure Databricks to configure the managed storage account's networking to deny all traffic except from trusted Azure services and your workspace subnets.
Reference: https://learn.microsoft.com/en-us/azure/databricks/security/network/classic/vnet-inject
Reference: https://learn.microsoft.com/en-us/azure/databricks/security/network/classic/secure-cluster-connectiv...
SOLUTION 3: COMPLETE PRIVATE LINK ISOLATION
For maximum security, Azure Databricks supports a "complete private isolation" configuration that combines multiple Private Link types. Based on your setup with VNet injection and NCC for serverless, here is the full picture:
1. Front-end (Inbound) Private Link: Secures user connections to the workspace UI and API. Create a private endpoint with target sub-resource "databricks_ui_api" in your transit VNet, plus a "browser_authentication" endpoint for SSO.
2. Classic Compute Plane (Back-end) Private Link: Secures the connection from your VNet-injected clusters to the Azure Databricks control plane. Create a private endpoint with target sub-resource "databricks_ui_api" in your workspace VNet.
3. NCC Private Endpoints for Serverless (which you already have): Secures serverless compute connections to your data resources.
4. Storage Service Endpoints or Private Endpoints: Ensures your classic compute reaches the workspace storage account privately, as described in Solutions 1 and 2 above.
When configuring complete private isolation:
- Set "Allow Public Network Access" to Disabled
- Set "Required NSG Rules" to NoAzureDatabricksRules
- You need all the endpoints listed above
Reference: https://learn.microsoft.com/en-us/azure/databricks/security/network/concepts/private-link
Reference: https://learn.microsoft.com/en-us/azure/databricks/security/network/classic/private-link-standard
BEST PRACTICE: MINIMIZE RELIANCE ON DBFS ROOT
While the above solutions address the networking challenge, the long-term best practice is to minimize your dependence on the workspace storage account altogether:
1. Use Unity Catalog with your own ADLS Gen2 storage account for all production data. Configure managed storage locations at the metastore, catalog, or schema level, pointing to storage accounts you fully control.
2. Disable DBFS root and mounts in your workspace settings once you have migrated all workflows to Unity Catalog volumes, external locations, or workspace files. This is available under Workspace Admin Settings, Security, "Disable DBFS root and mounts."
3. Even after disabling DBFS root user access, the workspace storage account still exists for internal system operations (notebook revisions, job results, etc.), but the volume of traffic is significantly reduced, and user data no longer flows through it.
Reference: https://learn.microsoft.com/en-us/azure/databricks/dbfs/dbfs-root
Reference: https://learn.microsoft.com/en-us/azure/databricks/dbfs/disable-dbfs-root-mounts
Reference: https://learn.microsoft.com/en-us/azure/databricks/connect/unity-catalog/cloud-storage/managed-stora...
ANSWERING YOUR SPECIFIC QUESTIONS
"Is there a way to work around this?" -- Yes. Add Azure Storage service endpoints to your workspace subnets, which allows your VNet-injected compute to reach the workspace storage account over the Azure backbone without public internet. You can then restrict the storage account to deny public access.
"Is it even possible to achieve full private network isolation in this scenario?" -- Yes, absolutely. Azure Databricks supports complete private isolation combining front-end Private Link, back-end Private Link, NCC for serverless, and storage service endpoints or storage firewall configuration. The documentation provides a comprehensive guide for achieving this.
I would also recommend opening a support ticket if you need hands-on assistance with the ARM template configuration for the storage firewall, as the Databricks support team can help ensure all the parameters are correctly set for your specific deployment.
Hope this helps! Let me know if you have follow-up questions.
* This reply used an agent system I built to research and draft this response based on the wide set of documentation I have available and previous memory. I personally review the draft for any obvious issues and for monitoring system reliability and update it when I detect any drift, but there is still a small chance that something is inaccurate, especially if you are experimenting with brand new features.
