All Product Platform Updates posts

Re: Azure Databricks: Required change in resource firewalls allowlisting Databricks serverless compu

ittzzmalind — Thu, 21 May 2026 17:28:20 GMT

Hi Team,

im facing difficulty in configuring NSP

Serverless Compute + NCC configration in account console + Private Endpoint (PE) to storage account is working as expected.

When Network security perimeter (NSP) is added and storage account (SA), in NSP we have 2 access mode to choose (transition and enforced )

1). Serverless + NCC + PE and Storage account (SA) Public access disabled - Working

2). Serverless + NCC + PE + NSP (either transition or enforced) and Storage account (SA) Public access disabled -working

ERROR :

3). Serverless + NCC + PE + NSP (either transition or enforced) and Storage account (SA) network access selected as Secured by perimeter - Not Working - Error

simple select query :

select * from databricks_training.training.employees returns error-
[UNAUTHORIZED_ACCESS] Unauthorized access: PERMISSION_DENIED: Request for user delegation key is not authorized. Details: None SQLSTATE: 42501
Note : In NSP inbound rule, there is option to select service tag, i have selected both global and regional serverless service tag.

My understanding is that the data plane communication should occur through the private endpoint. However, I am unsure whether any control plane communication is also being initiated and getting blocked, which could be causing the issue.

At this point, I do not have complete clarity on the exact root cause. Additionally, I anticipate that a similar issue may also occur with Classic Compute clusters in the same setup.

See attachment for configurations

Assistance Needed:

Is there any mitigation or supported configuration available when NSP is configured on a Storage Account and the storage account access level is set to “Secured by perimeter,” while accessing it from Databricks clusters?

Could you please help clarify how this scenario should be configured to avoid the access error?

Additionally:

Is the issue occurring because control plane communication is being blocked?
Since the “Allow Azure Databricks Control Plane” option via service tags is not available in this configuration, is there any alternative configuration or recommended approach that would work in this setup?

Azure Databricks: Required change in resource firewalls allowlisting Databricks serverless compute

AlexEsibov — Fri, 13 Mar 2026 19:21:11 GMT

To improve serverless scalability and expand service endpoint support to additional resources, Azure Databricks will make a change to the network identification used to restrict public access to storage access from serverless compute.

Re: Azure Databricks — Upgrade Managed Workspace to VNet Injected Workspace

KiranAnand — Thu, 15 Jan 2026 18:45:15 GMT

@carobitap, Yes, if you are sharing a VNet across multiple workspaces, you should use separate public and private subnets for each one. Each Databricks workspace requires two dedicated subnets (that cannot be shared) within a VNet.

Re: Azure Databricks — Upgrade Managed Workspace to VNet Injected Workspace

carobitap — Thu, 15 Jan 2026 17:07:13 GMT

If I create multiple vnet injected workspaces that share the same vnet, should I use different public/private subnets for each one?

Re: Azure Databricks — Upgrade Managed Workspace to VNet Injected Workspace

KiranAnand — Wed, 17 Dec 2025 19:55:30 GMT

@Raman_Unifeye, As you mentioned, the questions above are not related to the content in this blog post. I recommend that you direct these questions to your Azure Databricks account team for clarification.

Re: Azure Databricks — Upgrade Managed Workspace to VNet Injected Workspace

Raman_Unifeye — Wed, 17 Dec 2025 16:35:09 GMT

@KiranAnand - Thanks for the documentation.

The questions I have are around the 'Classic Compute' and it may not matter for the Managed/VNet-injected workspace.

1) VM Lifecycle management - Does the Azure Admin get any opportunity to contol the Azure VM in terms of allocation of 'same private IP address and hostname after termination/restart' ?

2) Any antivirus/endpoint protection/security baselines be injected into these 'ephemeral' VMs - like init script?

3) Any visibility does a customer have into underlying OS or runtime patching?

4) while looking to use 'serverless' for data processing in databricks, are there any Databricks scenarios (maintenance jobs etc) which will force to use classic compute?

Re: Azure Databricks — Upgrade Managed Workspace to VNet Injected Workspace

Griz — Wed, 17 Dec 2025 15:49:28 GMT

Generally NO, however, Microsoft may be able to reverse the operation with a support ticket. Since Databricks is a first party service on Azure, Microsoft would be the first level of support. Most customers who deploy the workspace with the "Easy" button (Managed VNet), may want to improve their security and/or route-ability, so they would use this procedure to move to a managed VNet and would not want to go back to managed VNet.

Re: Azure Databricks — Upgrade Managed Workspace to VNet Injected Workspace

sigmanudude — Wed, 17 Dec 2025 15:38:54 GMT

Thank you, is this operation reversable?

Re: Azure Databricks — Upgrade Managed Workspace to VNet Injected Workspace

Griz — Tue, 16 Dec 2025 14:36:31 GMT

Converting to a VNet injected workspace does NOT delete or remove any data @sigmanudude . The process only changes the networking of the deployed workspace.

Re: Azure Databricks — Upgrade Managed Workspace to VNet Injected Workspace

sigmanudude — Mon, 15 Dec 2025 23:22:57 GMT

does the redeployment delete or remove any of the catalogs or notebooks in the databricks environment?

Re: IP CIDR For Azure Databricks Hive Metastore

joycexudata — Wed, 12 Nov 2025 23:14:30 GMT

@nayan_wylde sorry for the late reply, yes the maintenance window is rescheduled. For up-to-date maintenance of the change, please check https://status.azuredatabricks.net/.

Re: IP CIDR For Azure Databricks Hive Metastore

nayan_wylde — Tue, 21 Oct 2025 15:22:03 GMT

I heard it is extended till Nov 6th... Is this true?

Re: IP CIDR For Azure Databricks Hive Metastore

joycexudata — Thu, 09 Oct 2025 22:47:45 GMT

@Rob_Hughes We expect only a small number of customers to be affected — specifically those who have hard-coded static IPs in their firewall or UDRs. This setup isn’t recommended, as explicitly called out in our public documentation. The blog post mainly serves as a maintenance notice and a record for future reference..

Re: IP CIDR For Azure Databricks Hive Metastore

Rob_Hughes — Mon, 06 Oct 2025 08:30:52 GMT

Is this for real ? Something posted on 1st Oct with a completion date of "Starting on mid of October" !!

Re: IP CIDR For Azure Databricks Hive Metastore

Khaja_Zaffer — Wed, 01 Oct 2025 14:37:19 GMT

Great information

IP CIDR For Azure Databricks Hive Metastore

joycexudata — Wed, 01 Oct 2025 14:01:12 GMT

Do you have hard code Hive Metastore IP address in your Azure user-defined routes or firewall appliance? Please read on.

What's new in Databricks - August 2025

Sujitha — Wed, 10 Sep 2025 14:08:13 GMT

Data Engineering

Migrate Lakeflow Declarative Pipelines from legacy publishing mode is GA

Lakeflow Declarative Pipelines has a legacy publishing mode that only allows publishing to a single catalog and schema. The default publishing mode enables publishing to multiple catalogs and schemas. 📖 Documentation

Selectively and atomically replace data with Insert replace using and insert replace on is GA

INSERT REPLACE USING replaces rows when the USING columns compare equal under equality. INSERT REPLACE ON replaces rows when they match a user-defined condition.

Microsoft SQL Server connector and ServiceNow are GA

Salesforce Data Cloud File Sharing connector is GA

New table property for Delta lake compression

You can explicitly set the compression codec for a Delta table using the delta.parquet.compression.codec table property. 📖 Documentation

Create external Delta tables from third party clients

You can now create Unity Catalog external tables backed by Delta Lake from external clients and systems, such as Apache Spark.📖 Documentation

Lakeflow Declarative Pipeline improvements

You can change the identity that a pipeline uses to run updates and the owner of tables published by the pipeline. This feature allows you to set a service principal as the run-as identity, which is safer and more reliable than using user accounts for automated workloads. 📖 Documentation

You can now easily create an ETL pipeline in a bundle in the workspace using the new Lakeflow Declarative Pipelines template project.

You can use automatic liquid clustering with CLUSTER BY AUTO and Databricks intelligently chooses clustering keys to optimize query performance.

⚡Lakebase

OLTP Database tab renamed to Lakebase Postgres

Lakebase synced tables supports syncing Apache Iceberg and Foreign tables

You can create synced tables in Snapshot sync mode from Iceberg tables or foreign tables. 📖 Documentation

Data type mapping

For new synced tables: TIMESTAMP types in source tables are mapped to TIMESTAMP WITH TIMEZONE in synced tables.

Budget policy is supported by Lakebase

You can tag a database instance and a synced table with a budget policy to attribute billing usage to specific policies. Additionally, custom tags can be added to a database instance for more granular attribution of compute usage to teams, projects, or cost centers.

Lakebase is enabled by default

The Lakebase: Managed Postgres OLTP Database preview is enabled by default

🪄Serverless

Serverless compute for notebooks, workflows, and Lakeflow Declarative Pipelines is available in the Asia Pacific (Jakarta) region (ap-southeast-3)

Base environments are custom environment specifications for serverless notebooks that define a serverless environment version and a set of dependencies. 📖 Documentation

Serverless GPU supports hyperparameters sweeps and Multinode Workloads and schedule jobs. 📖 Documentation

New features are available on Serverless Compute:

Databricks connect upgraded to 17.0
Scalar Python UDFS support service credentials
PySpark and Spark Connect support the df.mergeInto

🖥️Platform

Notebooks Improvements

Edit Mode in Assistant does multi-cell code refactoring and more. 📖Documentation.

Use the cell execution minimap to track your notebook’s progress at a glance. The minimap appears in the right margin and shows each cell’s execution state. Hover to see cell details, or click to jump directly to a cell.

Notebook autocomplete supports enhanced suggestions for complex data types including structs, maps, and arrays in SQL cells.

Lakeflow job improvements

Jobs that are set to run in continuous mode have the option to retry individual tasks on task failure. 📖Documentation

Power BI Databricks connector supports M2M OAuth

You can authenticate into Power BI Desktop using M2M OAuth. Databricks recommends switching to the new client credentials authentication option. 📖Documentation

Account SCIM 2.0 updates

Databricks has updated the Account SCIM API for identity management as follows:

Calling GET with filter params filter=displayName eq value_without_quotes results in a syntax error. To prevent this error, use quotation marks to wrap the value
Calling GET /api/2.0/accounts/{account_id}/scim/v2/Groups no longer returns members. Instead, iterate through get group details to get membership information
Calling PATCH /api/2.0/accounts/{account_id}/scim/v2/Groups/{id} returns a 204 response instead of a 200 response.

OAuth token federation is GA

Databricks assistant improvements

You can chat with Databricks Assistant on some compute pages. Use the Assistant chat panel to help you create a new compute resource, pool, and policy.

You can tailor how Databricks Assistant responds by adding custom user instructions. Guide the Assistant with preferences, coding conventions, and response guidelines.

Disable legacy features for new workspaces

A new account console setting allows account admins to disable certain legacy features on new workspaces created in their account. 📖Documentation

Serverless Workspaces are available

🤖GenAI & ML

AI Playground is GA

OpenAI GPT OSS models are available on Mosaic AI Model Serving

Mosaic AI Model Serving supports OpenAI's GPT OSS 120B and GPT OSS 20B as Databricks-hosted foundation models. 📖Documentation

Batch inference with GPT OSS models

OpenAI GPT OSS 120B and GPT OSS 20B are optimized for AI Functions, which means you can perform batch inference using these models and AI Functions like ai_query()

External MCP servers are in Beta

You can connect Databricks to external MCP servers. 📖 Documentation

Mosaic AI Vector Search reranker is available

Mosaic AI Vector Search offers reranking to help improve retrieval quality. 📖 Documentation

Token based rate limits are available on AI Gateway

You can configure token-based rate limits on your model serving endpoints.. 📖 Documentation

OpenAI GPT OSS models releases

The Databricks-hosted foundation models, OpenAI GPT OSS 120B and GPT OSS 20B support function and tool calling and provisioned throughput. 📖 Documentation

📝AIBI Genie

SQL expression validation for join relationships
File uploads are accessible only to the users who uploaded them
You can define join relationships locally within a Genie space's knowledge store. This is useful when authors lack permissions to define primary and foreign keys on upstream tables or when the join relationship has specific requirements, such as one-to-many or complex joins
Value dictionaries select the most frequent 1024 values from the first 100k rows, instead of the first 1024 values encountered.

📊AIBI Dashboard

Dashboard external embedding Public Preview and new custom calculations functions
You can define up to 200 custom calculations per dashboard.

🛡️Governance

Governed tags are public preview

You can create governed tags to enforce consistent tagging across data assets such as catalogs, schemas, and tables. Admins define the allowed keys and values and control which users and groups can assign them to objects. This helps standardize metadata for data classification, cost tracking, access control and automation. 📖Documentation

Single node compute on standard access mode is GA

This configuration allows multiple users to share a single-node compute resource with full user isolation. Single-node compute is useful for small jobs or non-distributed workloads.

Column masks retained when replacing a table

If a column in the new table matches a column name from the original table, its existing column mask is retained, even if no mask is specified. This change prevents accidental removal of column-level security policies during table replacement.

Access requests in Unity Catalog

You can enable self-service access requests in Unity Catalog by configuring access request destinations on securable objects.

Users can request access to Unity Catalog objects that they discover. These requests are sent to configured destinations, such as emails, Slack or Microsoft Teams channels, or they can be redirected to an internal access management system.

📖Documentation

Path credential vending

You can use path credential vending to grant short-lived credentials to external locations in your Unity Catalog metastore. Documentation

🔍 Data Warehousing

Default warehouse setting is available in Beta

Support for timestamp without time zone syntax

Support for schema and catalog level default collation

Expanded spatial SQL expressions and GEOMETRY and GEOGRAPHY data types

Re: Azure Databricks — Upgrade Managed Workspace to VNet Injected Workspace

Griz — Tue, 09 Sep 2025 16:19:34 GMT

thanks for documenting this procedure. I am sure this will help many customers who chose the "easy button" during deployment.

Azure Databricks — Upgrade Managed Workspace to VNet Injected Workspace

KiranAnand — Mon, 08 Sep 2025 17:19:32 GMT

Managed Workspace in Azure Databricks

Azure Databricks, a first-party service on Microsoft Azure, seamlessly integrates with other Azure services and is easily deployed through the Azure portal with a few clicks. This simplified deployment with basic configurations creates a “default deployment” or a managed Databricks workspace.

A managed Databricks workspace is a fully managed Azure service where Databricks creates and deploys a new Azure virtual network (VNet) to a locked managed resource group during workspace creation. All classic compute plane resources are associated with this managed VNet. The managed resource group containing the VNet is completely locked down, denying all actions except creating Private Endpoints to Storage Accounts. In short, Azure Databricks managed workspace, while convenient, lacks the security, customization, and control required for production, as well as long-term lower-tier and high-security environments.

Previously, Azure Databricks users had to create a new VNet injected workspace and migrate all workloads from the existing managed workspace to enable VNet injection. This process was necessary because there was no direct way to convert a managed workspace to a VNet injected workspace. However, a new preview feature now allows for the direct conversion of a managed workspace to a VNet injected workspace, eliminating the need for workspace creation and workload migration. This blog post will detail how to use this new feature to upgrade existing managed workspaces.

Prerequisites

You must NOT use a Managed VNet with an egress load balancer (LB).
All clusters and warehouses (interactive and jobs) in Classic compute must be terminated before applying the update. The best practice would be to bring down and terminate all the compute (Classic and Serverless).
New clusters (interactive and jobs) cannot be started during the update.

Limitations and Other Considerations

Upgrading a workspace to VNet injected needs to be done as a singular atomic operation. No other changes to the workspace are supported while performing this operation. If you want to change any other network settings of the workspace, like enabling Secure Cluster Connectivity or adding private endpoints, those changes need to take place as a subsequent separate operation, after the workspace is upgraded to VNet injection.
Terraform is not yet supported on this feature.
Any workspace deployed with a Managed VNet would have a “workers-vnet” created in the Managed Resource Group by Databricks. This VNet has an address space of 65K IPs. Two subnets are created within this VNet with an address space of 16K IPs. So, when converting such a workspace from Managed VNet to VNet Injection, proper care should be taken to size the new VNet and the corresponding subnets. Please reach out to your Databricks account team if you need further assistance with this process.

Convert a managed workspace to a VNet Injected workspace

Now, we will demonstrate a new feature that recently became available as a Preview in Azure Databricks to convert a managed workspace into a VNet injected one. We will use the Azure web portal to perform such a conversion in this blog. This section has multiple steps that are expected to be performed in that order. In case you have already created the required Network Security Group (NSG), VNet, and subnets, you can use those existing ones and skip the steps to create them.

Step-1 Create Network Security Group

Create a Network Security Group (NSG) in Azure by following this official documentation from Microsoft.

Note: The NSG must be in the same Azure region and Azure subscription as the Databricks workspace.

Once it is created, note down the name of the NSG, and that would be required in the subsequent step.

Step-2 Create Virtual Network

As the next step, create a custom virtual network (VNet) and two subnets as needed by Azure Databricks workspaces. Refer to this official documentation from Microsoft for more details. Please note that Azure Databricks requires two subnets for its use. Databricks provides detailed specifications of the VNet and subnet requirements in their official documentation. Importantly, one of the prerequisites for the upgrade of a managed workspace to VNet injection is that the subnets used should be delegated to Databricks. More details on how to do this are specified below. If you have already created the required VNet and subnets with all the requirements, you can skip this step and use those existing ones.

Note: The VNet must be in the same Azure region and Azure subscription as the Databricks workspace.

While adding each of the subnets to the VNet, you need to make sure that you select the following two options. A screenshot is provided below for quick reference.

In the “Security” section of the subnet creation, select the name of the NSG created in the previous step in the dropdown option for “Network security group” for both the subnets.
In the “Subnet Delegation” section, select the value as “Microsoft.Databricks/workspaces” from the dropdown for both the subnets.

Note: To keep this blog simple, we are not getting into the details of adding an explicit outbound access and a stable egress public IP address using a NAT gateway to the VNet.

While deploying the VNet, you can choose any explicit outbound access model that suits your organization’s needs, such as Azure Firewall, Network Virtual Appliances (NVAs), or NAT Gateways.

Warning: This official communication from Microsoft suggests that on September 30, 2025, default outbound access connectivity for virtual machines in Azure will be retired. This means that Azure Databricks workspaces that use default outbound access rather than a stable egress public IP might not continue to work after that date. We recommend that you add explicit outbound methods, like a NAT gateway or some other model specified above for your workspaces, before that date.

After the VNet is created, note down the resource ID of this VNet as suggested below. Go to the VNet that you just created and choose the “JSON View”, and you will find the resource ID of the VNet as the first entry. Copy this value and save it for use in the next step.

Step-3 Upgrade the Workspace to VNet Injection

In this process, we will update the properties of the current Databricks workspace to perform a VNet injection. We will generate the workspace template for the current workspace, modify it to include the necessary changes, and then save it, which will force a redeployment of the workspace.

Note: All clusters (interactive and jobs) must be terminated before applying the update.

As the first step, select the Databricks workspace overview page in the Azure portal and then choose the “Export Template” option, under the automation section on the left navigation.

Once the “Generating template……” screen is complete, select the “Deploy” option.

And then select the “Edit template” option to make the necessary changes.

The details of the changes are listed below. Also included below is a screenshot of the changes.

Update the “apiVersion” in the template to be “2025-08-01-preview”
Remove the following fields from the template if they exist
- “vnetAddressPrefix”
- “natGatewayName”
- “publicIpName”
Add the fields below to the template
- A sample JSON block to add your subnets and VNet to the workspace template is given below. Replace the values in the angular brackets with the proper ones, and then add this section to the workspace template under the “Resources >> properties >> parameters”. Make sure that JSON has commas in the right places and is valid after the edits.
- Use the details of the VNet you just created in the previous step or the VNet you want to use.

"customPrivateSubnetName": {
"value": "<Your private subnet name>"
},
"customPublicSubnetName": {
"value": "<Your public subnet name>"
},
"customVirtualNetworkId": {
"value": "<Resource ID of your VNet>"
}

Ensure that all the other fields in the template are left as is without any changes

Save the template after making the changes as suggested above, and then select “Review + create”, and then “Create” in the subsequent screens. This will force a redeployment of the workspace with the new networking parameters and convert it into a VNet injected workspace.

Conclusion

To summarize, the managed workspace of Azure Databricks is not an ideal choice in most usage scenarios, especially in long-lived environments. Till now, to get off of an Azure Databricks managed workspace, the only option was to create a new VNet injected workspace and migrate everything from the managed workspace to it. But with the release of this new feature, customers have an easier option to upgrade a managed workspace to a VNet injected workspace. In this blog, we saw how easy it is to upgrade an existing managed workspace in Azure Databricks to a VNet injected one, using the newly available preview feature.

Re: What's new in Databricks: July - August 2025

TheOC — Wed, 03 Sep 2025 09:34:25 GMT

That's... a lot!
It's really cool to use a product that is moving so quickly. Excited to give all these changes / features a closer look. Thanks @Sujitha !