Managed workspace in Azure Databricks
Azure Databricks, a first-party service on Microsoft Azure, seamlessly integrates with other Azure services and is easily deployed through the Azure portal with a few clicks. This simplified deployment with basic configurations creates a “default deployment” or a managed Databricks workspace.
A managed Databricks workspace is a fully managed Azure service where Databricks creates and deploys a new Azure virtual network (VNet) to a locked managed resource group during workspace creation. All classic compute plane resources are associated with this managed VNet. The managed resource group containing the VNet is completely locked down, denying all actions except for the creation of Private Endpoints to Storage Accounts. In short, Azure Databricks managed workspace, while convenient, lacks the security, customization, and control required for production, long-term lower-tier, and high-security environments.
Previously, Azure Databricks users had to create a new VNet injected workspace and migrate all workloads from the existing managed workspace to enable VNet injection. This process was necessary because there was no direct way to convert a managed workspace to a VNet injected workspace. However, a new preview feature now allows for the direct conversion of a managed workspace to a VNet injected workspace, eliminating the need for workspace creation and workload migration. This blog post will detail how to use this new feature to upgrade existing managed workspaces.
Prerequisites
- Enroll in the preview of this feature through your Databricks account team.
- Provide your Azure Subscription ID for the workspaces that need to be enabled for this preview.
- You must NOT be using a Managed VNet with egress load balancer (LB).
- All clusters and warehouses (interactive and jobs) in Classic compute must be terminated before applying the update. The best practice would be bring down and terminate all the compute (Classic and Serverless).
- New clusters (interactive and jobs) cannot be started during the update.
Limitations and Other Considerations
- Upgrading a workspace to VNet injected needs to be done as a singular atomic operation. No other changes to the workspace are supported while performing this operation. If you want to change any other network settings of the workspace, like enabling Secure Cluster Connectivity or add private endpoints, those changes need to take place as a subsequent separate operation, after the workspace is upgraded to VNet injection.
- Terraform is not yet supported on this feature.
- Any workspace deployed with a Managed VNet would have a “workers-vnet” created in the Managed Resource Group by Databricks. This VNet has an address space of 65K IPs. Also, there are two subnets created within this VNet with an address space of 16K IPs. So when converting such a workspace from Managed VNet to VNet Injection, proper care should be taken to size the new VNet and the corresponding subnets. Please reach out to your Databricks account team if you need further assistance with this process.
Convert a managed workspace to a VNet Injected workspace
Now, we will demonstrate a new feature that recently became available as a Preview in Azure Databricks to convert a managed workspace into a VNet injected one. We will use the Azure web portal to perform such a conversion in this blog. This section has multiple steps that are expected to be performed in that order. In case you have already created the required Network Security Group (NSG), VNet, and subnets, you can use those existing ones and skip the steps to create them.
Step-1 Create Network Security Group
Create a Network Security Group (NSG) in Azure by following this official documentation from Microsoft.
Note: The NSG must be in the same Azure region and Azure subscription as the Databricks workspace.
Once it is created, note down the name of the NSG, and that would be required in the subsequent step.
Step-2 Create Virtual Network
As the next step, create a custom virtual network (VNet) and two subnets as needed by Azure Databricks workspaces. Refer to this official documentation from Microsoft for more details. Please note that Azure Databricks requires two subnets for its use. Databricks provides detailed specifications of the VNet and subnet requirements in their official documentation. Importantly, one of the prerequisites for the upgrade of a managed workspace to VNet injection is that the subnets used should be delegated to Databricks. More details on how to do this are specified below. If you have already created the required VNet and subnets with all the requirements, you can skip this step and use those existing ones.
Note: The VNet must be in the same Azure region and Azure subscription as the Databricks workspace.
While adding each of the subnets to the VNet, you need to make sure that you select the following two options. A screenshot is provided below for quick reference.
- In the “Security” section of the subnet creation, select the name of the NSG created in the previous step in the dropdown option for “Network security group” for both the subnets.
- In the “Subnet Delegation” section, select the value as “Microsoft.Databricks/workspaces” from the dropdown for both the subnets.
Note: To keep this blog simple, we are not getting into the details of adding an explicit outbound access and a stable egress public IP address using a NAT gateway to the VNet.
While deploying the VNet, you can choose any explicit outbound access model that suits your organization’s needs such as Azure Firewall, Network Virtual Appliances (NVAs), or NAT Gateways.
Warning: This official communication from Microsoft suggests that on September 30, 2025, default outbound access connectivity for virtual machines in Azure will be retired. This means that Azure Databricks workspaces that use default outbound access rather than a stable egress public IP might not continue to work after that date. We recommend that you add explicit outbound methods, like a NAT gateway or some other model specified above for your workspaces, before that date.
After the VNet is created, note down the resource ID of this VNet as suggested below. Go to the VNet that you just created and choose the “JSON View”, and you will find the resource ID of the VNet as the first entry. Copy this value and save it for use in the next step.
Step-3 Upgrade the Workspace to VNet Injection
In this process, we will update the properties of the current Databricks workspace to perform a VNet injection. We will generate the workspace template for the current workspace, modify it to include the necessary changes, and then save it back, which will force a redeployment of the workspace.
Note: All clusters (interactive and jobs) must be terminated before applying the update.
As the first step, select the Databricks workspace overview page in the Azure portal and then choose the “Export Template” option, under the automation section on the left navigation.
Once the “Generating template……” screen is complete, select the “Deploy” option.
And then select the “Edit template” option to make the necessary changes.
The details of the changes are listed below. Also included below is a screenshot of the changes.
- Update the “apiVersion” in the template to be “2025–02–01-preview”
- Remove the following fields from the template if they exist
- “vnetAddressPrefix”
- “natGatewayName”
- “publicIpName” - Add the below fields in the template
- A sample JSON block to add your subnets and VNet to the workspace template is given below. Replace the values in the angular brackets with the proper ones, and then add this section to the workspace template under the “Resources >> properties >> parameters”. Make sure that JSON has commas in the right places, and is valid after the edits.
- Use the details of the VNet you just created in the previous step or the VNet you want to use.
"customPrivateSubnetName": {
"value": "<Your private subnet name>"
},
"customPublicSubnetName": {
"value": "<Your public subnet name>"
},
"customVirtualNetworkId": {
"value": "<Resource ID of your VNet>"
}- Ensure that all the other fields in the template are left as is without any changes
Save the template after making the changes as suggested above, and then select “Review + create”, and then “Create” in the subsequent screens. This will force a redeployment of the workspace with the new networking parameters and convert it into a VNet injected workspace.
Conclusion
To summarize, the managed workspace of Azure Databricks is not an ideal choice in most usage scenarios, especially in long-lived environments. Till now, to get off of an Azure Databricks managed workspace, the only option was to create a new VNet injected workspace and migrate everything from the managed workspace to it. But with the release of this new feature, customers have an easier option to upgrade a managed workspace to a VNet injected workspace. In this blog, we saw how easy it is to upgrade an existing managed workspace in Azure Databricks to a VNet injected one, using the newly available preview feature.
