Product Platform Updates articles

Azure Databricks: Required change in resource firewalls allowlisting Databricks serverless compute

AlexEsibov — Fri, 13 Mar 2026 19:21:11 GMT

To improve serverless scalability and expand service endpoint support to additional resources, Azure Databricks will make a change to the network identification used to restrict public access to storage access from serverless compute.

IP CIDR For Azure Databricks Hive Metastore

joycexudata — Wed, 01 Oct 2025 14:01:12 GMT

Do you have hard code Hive Metastore IP address in your Azure user-defined routes or firewall appliance? Please read on.

What's new in Databricks - August 2025

Sujitha — Wed, 10 Sep 2025 14:08:13 GMT

Data Engineering

Migrate Lakeflow Declarative Pipelines from legacy publishing mode is GA

Lakeflow Declarative Pipelines has a legacy publishing mode that only allows publishing to a single catalog and schema. The default publishing mode enables publishing to multiple catalogs and schemas. 📖 Documentation

Selectively and atomically replace data with Insert replace using and insert replace on is GA

INSERT REPLACE USING replaces rows when the USING columns compare equal under equality. INSERT REPLACE ON replaces rows when they match a user-defined condition.

Microsoft SQL Server connector and ServiceNow are GA

Salesforce Data Cloud File Sharing connector is GA

New table property for Delta lake compression

You can explicitly set the compression codec for a Delta table using the delta.parquet.compression.codec table property. 📖 Documentation

Create external Delta tables from third party clients

You can now create Unity Catalog external tables backed by Delta Lake from external clients and systems, such as Apache Spark.📖 Documentation

Lakeflow Declarative Pipeline improvements

You can change the identity that a pipeline uses to run updates and the owner of tables published by the pipeline. This feature allows you to set a service principal as the run-as identity, which is safer and more reliable than using user accounts for automated workloads. 📖 Documentation

You can now easily create an ETL pipeline in a bundle in the workspace using the new Lakeflow Declarative Pipelines template project.

You can use automatic liquid clustering with CLUSTER BY AUTO and Databricks intelligently chooses clustering keys to optimize query performance.

⚡Lakebase

OLTP Database tab renamed to Lakebase Postgres

Lakebase synced tables supports syncing Apache Iceberg and Foreign tables

You can create synced tables in Snapshot sync mode from Iceberg tables or foreign tables. 📖 Documentation

Data type mapping

For new synced tables: TIMESTAMP types in source tables are mapped to TIMESTAMP WITH TIMEZONE in synced tables.

Budget policy is supported by Lakebase

You can tag a database instance and a synced table with a budget policy to attribute billing usage to specific policies. Additionally, custom tags can be added to a database instance for more granular attribution of compute usage to teams, projects, or cost centers.

Lakebase is enabled by default

The Lakebase: Managed Postgres OLTP Database preview is enabled by default

🪄Serverless

Serverless compute for notebooks, workflows, and Lakeflow Declarative Pipelines is available in the Asia Pacific (Jakarta) region (ap-southeast-3)

Base environments are custom environment specifications for serverless notebooks that define a serverless environment version and a set of dependencies. 📖 Documentation

Serverless GPU supports hyperparameters sweeps and Multinode Workloads and schedule jobs. 📖 Documentation

New features are available on Serverless Compute:

Databricks connect upgraded to 17.0
Scalar Python UDFS support service credentials
PySpark and Spark Connect support the df.mergeInto

🖥️Platform

Notebooks Improvements

Edit Mode in Assistant does multi-cell code refactoring and more. 📖Documentation.

Use the cell execution minimap to track your notebook’s progress at a glance. The minimap appears in the right margin and shows each cell’s execution state. Hover to see cell details, or click to jump directly to a cell.

Notebook autocomplete supports enhanced suggestions for complex data types including structs, maps, and arrays in SQL cells.

Lakeflow job improvements

Jobs that are set to run in continuous mode have the option to retry individual tasks on task failure. 📖Documentation

Power BI Databricks connector supports M2M OAuth

You can authenticate into Power BI Desktop using M2M OAuth. Databricks recommends switching to the new client credentials authentication option. 📖Documentation

Account SCIM 2.0 updates

Databricks has updated the Account SCIM API for identity management as follows:

Calling GET with filter params filter=displayName eq value_without_quotes results in a syntax error. To prevent this error, use quotation marks to wrap the value
Calling GET /api/2.0/accounts/{account_id}/scim/v2/Groups no longer returns members. Instead, iterate through get group details to get membership information
Calling PATCH /api/2.0/accounts/{account_id}/scim/v2/Groups/{id} returns a 204 response instead of a 200 response.

OAuth token federation is GA

Databricks assistant improvements

You can chat with Databricks Assistant on some compute pages. Use the Assistant chat panel to help you create a new compute resource, pool, and policy.

You can tailor how Databricks Assistant responds by adding custom user instructions. Guide the Assistant with preferences, coding conventions, and response guidelines.

Disable legacy features for new workspaces

A new account console setting allows account admins to disable certain legacy features on new workspaces created in their account. 📖Documentation

Serverless Workspaces are available

🤖GenAI & ML

AI Playground is GA

OpenAI GPT OSS models are available on Mosaic AI Model Serving

Mosaic AI Model Serving supports OpenAI's GPT OSS 120B and GPT OSS 20B as Databricks-hosted foundation models. 📖Documentation

Batch inference with GPT OSS models

OpenAI GPT OSS 120B and GPT OSS 20B are optimized for AI Functions, which means you can perform batch inference using these models and AI Functions like ai_query()

External MCP servers are in Beta

You can connect Databricks to external MCP servers. 📖 Documentation

Mosaic AI Vector Search reranker is available

Mosaic AI Vector Search offers reranking to help improve retrieval quality. 📖 Documentation

Token based rate limits are available on AI Gateway

You can configure token-based rate limits on your model serving endpoints.. 📖 Documentation

OpenAI GPT OSS models releases

The Databricks-hosted foundation models, OpenAI GPT OSS 120B and GPT OSS 20B support function and tool calling and provisioned throughput. 📖 Documentation

📝AIBI Genie

SQL expression validation for join relationships
File uploads are accessible only to the users who uploaded them
You can define join relationships locally within a Genie space's knowledge store. This is useful when authors lack permissions to define primary and foreign keys on upstream tables or when the join relationship has specific requirements, such as one-to-many or complex joins
Value dictionaries select the most frequent 1024 values from the first 100k rows, instead of the first 1024 values encountered.

📊AIBI Dashboard

Dashboard external embedding Public Preview and new custom calculations functions
You can define up to 200 custom calculations per dashboard.

🛡️Governance

Governed tags are public preview

You can create governed tags to enforce consistent tagging across data assets such as catalogs, schemas, and tables. Admins define the allowed keys and values and control which users and groups can assign them to objects. This helps standardize metadata for data classification, cost tracking, access control and automation. 📖Documentation

Single node compute on standard access mode is GA

This configuration allows multiple users to share a single-node compute resource with full user isolation. Single-node compute is useful for small jobs or non-distributed workloads.

Column masks retained when replacing a table

If a column in the new table matches a column name from the original table, its existing column mask is retained, even if no mask is specified. This change prevents accidental removal of column-level security policies during table replacement.

Access requests in Unity Catalog

You can enable self-service access requests in Unity Catalog by configuring access request destinations on securable objects.

Users can request access to Unity Catalog objects that they discover. These requests are sent to configured destinations, such as emails, Slack or Microsoft Teams channels, or they can be redirected to an internal access management system.

📖Documentation

Path credential vending

You can use path credential vending to grant short-lived credentials to external locations in your Unity Catalog metastore. Documentation

🔍 Data Warehousing

Default warehouse setting is available in Beta

Support for timestamp without time zone syntax

Support for schema and catalog level default collation

Expanded spatial SQL expressions and GEOMETRY and GEOGRAPHY data types

Azure Databricks — Upgrade Managed Workspace to VNet Injected Workspace

KiranAnand — Mon, 08 Sep 2025 17:19:32 GMT

Managed Workspace in Azure Databricks

Azure Databricks, a first-party service on Microsoft Azure, seamlessly integrates with other Azure services and is easily deployed through the Azure portal with a few clicks. This simplified deployment with basic configurations creates a “default deployment” or a managed Databricks workspace.

A managed Databricks workspace is a fully managed Azure service where Databricks creates and deploys a new Azure virtual network (VNet) to a locked managed resource group during workspace creation. All classic compute plane resources are associated with this managed VNet. The managed resource group containing the VNet is completely locked down, denying all actions except creating Private Endpoints to Storage Accounts. In short, Azure Databricks managed workspace, while convenient, lacks the security, customization, and control required for production, as well as long-term lower-tier and high-security environments.

Previously, Azure Databricks users had to create a new VNet injected workspace and migrate all workloads from the existing managed workspace to enable VNet injection. This process was necessary because there was no direct way to convert a managed workspace to a VNet injected workspace. However, a new preview feature now allows for the direct conversion of a managed workspace to a VNet injected workspace, eliminating the need for workspace creation and workload migration. This blog post will detail how to use this new feature to upgrade existing managed workspaces.

Prerequisites

You must NOT use a Managed VNet with an egress load balancer (LB).
All clusters and warehouses (interactive and jobs) in Classic compute must be terminated before applying the update. The best practice would be to bring down and terminate all the compute (Classic and Serverless).
New clusters (interactive and jobs) cannot be started during the update.

Limitations and Other Considerations

Upgrading a workspace to VNet injected needs to be done as a singular atomic operation. No other changes to the workspace are supported while performing this operation. If you want to change any other network settings of the workspace, like enabling Secure Cluster Connectivity or adding private endpoints, those changes need to take place as a subsequent separate operation, after the workspace is upgraded to VNet injection.
Terraform is not yet supported on this feature.
Any workspace deployed with a Managed VNet would have a “workers-vnet” created in the Managed Resource Group by Databricks. This VNet has an address space of 65K IPs. Two subnets are created within this VNet with an address space of 16K IPs. So, when converting such a workspace from Managed VNet to VNet Injection, proper care should be taken to size the new VNet and the corresponding subnets. Please reach out to your Databricks account team if you need further assistance with this process.

Convert a managed workspace to a VNet Injected workspace

Now, we will demonstrate a new feature that recently became available as a Preview in Azure Databricks to convert a managed workspace into a VNet injected one. We will use the Azure web portal to perform such a conversion in this blog. This section has multiple steps that are expected to be performed in that order. In case you have already created the required Network Security Group (NSG), VNet, and subnets, you can use those existing ones and skip the steps to create them.

Step-1 Create Network Security Group

Create a Network Security Group (NSG) in Azure by following this official documentation from Microsoft.

Note: The NSG must be in the same Azure region and Azure subscription as the Databricks workspace.

Once it is created, note down the name of the NSG, and that would be required in the subsequent step.

Step-2 Create Virtual Network

As the next step, create a custom virtual network (VNet) and two subnets as needed by Azure Databricks workspaces. Refer to this official documentation from Microsoft for more details. Please note that Azure Databricks requires two subnets for its use. Databricks provides detailed specifications of the VNet and subnet requirements in their official documentation. Importantly, one of the prerequisites for the upgrade of a managed workspace to VNet injection is that the subnets used should be delegated to Databricks. More details on how to do this are specified below. If you have already created the required VNet and subnets with all the requirements, you can skip this step and use those existing ones.

Note: The VNet must be in the same Azure region and Azure subscription as the Databricks workspace.

While adding each of the subnets to the VNet, you need to make sure that you select the following two options. A screenshot is provided below for quick reference.

In the “Security” section of the subnet creation, select the name of the NSG created in the previous step in the dropdown option for “Network security group” for both the subnets.
In the “Subnet Delegation” section, select the value as “Microsoft.Databricks/workspaces” from the dropdown for both the subnets.

Note: To keep this blog simple, we are not getting into the details of adding an explicit outbound access and a stable egress public IP address using a NAT gateway to the VNet.

While deploying the VNet, you can choose any explicit outbound access model that suits your organization’s needs, such as Azure Firewall, Network Virtual Appliances (NVAs), or NAT Gateways.

Warning: This official communication from Microsoft suggests that on September 30, 2025, default outbound access connectivity for virtual machines in Azure will be retired. This means that Azure Databricks workspaces that use default outbound access rather than a stable egress public IP might not continue to work after that date. We recommend that you add explicit outbound methods, like a NAT gateway or some other model specified above for your workspaces, before that date.

After the VNet is created, note down the resource ID of this VNet as suggested below. Go to the VNet that you just created and choose the “JSON View”, and you will find the resource ID of the VNet as the first entry. Copy this value and save it for use in the next step.

Step-3 Upgrade the Workspace to VNet Injection

In this process, we will update the properties of the current Databricks workspace to perform a VNet injection. We will generate the workspace template for the current workspace, modify it to include the necessary changes, and then save it, which will force a redeployment of the workspace.

Note: All clusters (interactive and jobs) must be terminated before applying the update.

As the first step, select the Databricks workspace overview page in the Azure portal and then choose the “Export Template” option, under the automation section on the left navigation.

Once the “Generating template……” screen is complete, select the “Deploy” option.

And then select the “Edit template” option to make the necessary changes.

The details of the changes are listed below. Also included below is a screenshot of the changes.

Update the “apiVersion” in the template to be “2025-08-01-preview”
Remove the following fields from the template if they exist
- “vnetAddressPrefix”
- “natGatewayName”
- “publicIpName”
Add the fields below to the template
- A sample JSON block to add your subnets and VNet to the workspace template is given below. Replace the values in the angular brackets with the proper ones, and then add this section to the workspace template under the “Resources >> properties >> parameters”. Make sure that JSON has commas in the right places and is valid after the edits.
- Use the details of the VNet you just created in the previous step or the VNet you want to use.

"customPrivateSubnetName": {
"value": "<Your private subnet name>"
},
"customPublicSubnetName": {
"value": "<Your public subnet name>"
},
"customVirtualNetworkId": {
"value": "<Resource ID of your VNet>"
}

Ensure that all the other fields in the template are left as is without any changes

Save the template after making the changes as suggested above, and then select “Review + create”, and then “Create” in the subsequent screens. This will force a redeployment of the workspace with the new networking parameters and convert it into a VNet injected workspace.

Conclusion

To summarize, the managed workspace of Azure Databricks is not an ideal choice in most usage scenarios, especially in long-lived environments. Till now, to get off of an Azure Databricks managed workspace, the only option was to create a new VNet injected workspace and migrate everything from the managed workspace to it. But with the release of this new feature, customers have an easier option to upgrade a managed workspace to a VNet injected workspace. In this blog, we saw how easy it is to upgrade an existing managed workspace in Azure Databricks to a VNet injected one, using the newly available preview feature.

What's new in Databricks: July 2025

Sujitha — Wed, 10 Sep 2025 14:08:37 GMT

July 2025 Release Highlights

🛠️Data Engineering

Moving tables between the Lakeflow declarative pipeline is GA

Teams can seamlessly reorganize streaming tables and materialized views as needs change—split a pipeline, consolidate workloads, or migrate to improved refresh schedules—without disruption or rebuilding from scratch. This increases agility in managing and scaling data platforms, helping maintain velocity as business requirements evolve. 📖 Documentation

Dynamic Partition overwrite with INSERT REPLACE USING

You can now surgically replace rows in a table based on match conditions, improving performance and simplifying updates out of the box. 📖 Documentation

Real-time mode is now available in Structured Streaming

You can now use Real Time mode, a trigger type for Structured Streaming that enables sub-second latency data processing. It’s designed for operational workloads that require immediate response to streaming data (typically from kafka to kafka). 📖 Documentation

🪄Serverless

New features are now available on Serverless Compute:

SQL procedure support
Set a default collation for SQL Functions
Recursive common table expressions (rCTE) support
PySpark and Spark Connect now support the DataFrames df.mergeInto API
Support ALL CATALOGS in SHOW SCHEMAS
Liquid clustering now compacts deletion vectors more efficiently
Allow non-deterministic expressions in UPDATE/INSERT column values for MERGE operations
Change Delta MERGE Python APIs to return DataFrame instead of Unit

⚡Lakebase

Synced tables are now automatically metered and billed

Synced tables are automatically tracked for usage and costs. This brings full cost transparency and control—critical for data teams scaling to many pipelines or sharing resources across business units. No more manual tracking when optimizing spend, forecasting budgets, or allocating costs between departments—usage is visible and billable out of the box. All usage is tracked and reported in the system.billing.usage

Databricks Apps support for Lakebase resources

Teams can now integrate Lakebase databases directly as resources for Databricks Apps, streamlining the workflow from analytics and operational data to application development. This enables building smarter, data-driven applications faster—without managing separate infrastructure or extra integrations—accelerating time to insight and innovation. 📖Documentation

Run job and for each have a separate limit

Complex workflows that rely on “Run job” or “For each” tasks can now operate in parallel without hitting the overall task concurrency ceiling. This means increased throughput for orchestrating batch workloads or parallel pipelines, less bottlenecking, and more efficient use of compute—especially in large-scale environments where maximizing concurrency is critical to meeting SLAs and deadlines.

🔗Data sharing

Delta Sharing supports sharing tables and schemas secured by ABAC

Delta Sharing providers can add tables and schemas secured by attribute-based access control to a Delta share. The policy does not govern the recipient's access, so recipients have full access to the shared asset. Recipients can apply their own ABAC policies.

The name of the organization is required to enable Delta Sharing on the metastore

An organization name must be specified if you are sharing data with a Databricks recipient not in your account when enabling Delta Sharing on your metastore. 📖Documentation

Git support updates

Databricks Git folder can be used to track and manage changes to alerts. 📖Documentation
You can use the UI to add and manage multiple Git credentials in the workspace from one or multiple git providers. 📖Documentation

Databricks Connector for Power BI supports the ADBC Driver

You can set the Databricks Connector for Power BI to use the Arrow Database Connectivity instead of the ODBC driver. 📖Documentation

🖥️Platform

New compute policy form

Jobs and Pipelines list now includes DBSQL Pipelines

The Jobs & Pipelines list now includes pipelines for materialized views and streaming tables that were created with Databricks SQL.

System tables improvements

The usage_metadata.job_name value in the system.billing.usage table now contains the run names for runs triggered through the one-time run API.

New columns are now available in the query history system table, providing additional query insights:

cache_origin_statement_id: For query results fetched from cache, this field contains the statement ID of the query that originally inserted the result into the cache.
query_parameters: A struct containing named and positional parameters used in parameterized queries.
written_rows: The number of rows of persistent data written to cloud object storage.
written_files: Number of files of persistent data written to cloud object storage.

Disable DBFS root and mounts is now available

You can disable access to DBFS and Mounts in existing Databricks workspaces. 📖Documentation

Notebooks improvements

Add a split view to edit notebooks side by side. 📖Documentation.
Pressing Cmd + F (Mac) or Ctrl + F (Windows) in a notebook now opens the native Databricks find-and-replace tool. This allows you to quickly search and replace text throughout your entire notebook, including content outside the current viewport. 📖Documentation

Data exploration using an LLM

You can ask natural language questions about the sample data using Catalog Explorer. The Assistant generates the SQL based on metadata context and table usage patterns. After the query is generated, you can validate the query and then run it against the underlying table.

Restore Python Variables after idle termination in Serverless Notebooks

Databricks snapshots your notebook’s Python variables before terminating idle serverless compute. When you reconnect your notebook is automatically restored from its snapshot letting you continue your work seamlessly.

🤖GenAI & ML

Expanding region availability for ai_parse_document

`ai_parse_document` is now available in the following regions (AWS):

`us-west-2/ us-east-1/ us-east-2 /ap-northeast-1 /ap-northeast-2 /ap-south-1 /ap-southeast-1`

`ap-southeast-2/ca-central-1/eu-central-1 /eu-west-1 /eu-west-2 /sa-east-1`

Agent Bricks: Multi-Agent Supervisor is in Beta

Agent Bricks: Multi-Agent Supervisor supports designing a multi-agent AI system that combines Genie spaces and Knowledge Assistant agent endpoints to work together on complex tasks that require different specialized skills. Documentation

Google Gemma 3 12B is available on Mosaic AI Model Serving

Gemma 3 12B supports text inputs for foundation models APIs pay-per-token and provisioned throughput and AI functions.

📝AIBI Genie

Caching enhancements speed up question performance when starting a new Genie session.
Genie automatically cancels query executions that take longer than 15 minutes to execute.
Genie allows editable parameters on date and numeric fields, expanding beyond previous support for string fields.
You can edit Genie space descriptions directly in place.

📊AIBI Dashboard

Authors can now undo and redo actions on the canvas using keyboard shortcuts or the toolbar.
Within the Add data dialog, authors can now filter to show only metric views.
Custom calculations now support AGGREGATE OVER to enable window function-like behavior, such as computing moving averages and running totals. 📖Documentation
The maximum number of pages per dashboard has increased from ten to fifteen.
Support for parameterized schedules has been restored. 📖Documentation

🛡️Governance

Unity Catalog Python User-defined Table functions

You can now register Python UDTFs in Unity Catalog for centralized governance and reusable logic across SQL queries. 📖Documentation

Scalar Python UDFs support service credentials

Scalar Python UDFs can use Unity Catalog service credentials to securely access external cloud services. This is useful for integrating operations such as cloud-based tokenization, encryption, or secret management directly into your data transformations.

🔍 Data Warehousing

IDENTIFIER support is now available in Databricks SQL for catalog operations.

CAN VIEW permission on SQL Warehouse is GA

The permission allows users to view SQL Warehouse, query history, and query profiles.

Expanded Spatial SQL expressions and Geometry and Geography Data types

Support for advanced spatial analytics and collation at the schema/catalog level simplifies complex location-based analytics and ensures consistency across all database objects, reducing friction in data warehousing projects. More than 80 new spatial SQL expressions are available. 📖Documentation

Support for Schema and Catalog level default collation

You can set a default collation for schemas and catalogs in Databricks Runtime 17.1. This allows you to define a collation that applies to all objects created within the schema or catalog, ensuring consistent collation behavior across your data.

Databricks on AWS: Opening new ports for Databricks

GlorianaHM — Mon, 01 Sep 2025 06:31:47 GMT

To improve scalability and enable new features, Databricks classic compute plane will start communicating to the Databricks control plane over 10 new ports: 3306 and 8443-8451. You may need to update your network configurations before we start using these ports on July 22, 2025.

Please ensure that the cloud administrator responsible for managing port configurations in your organization reviews the information below.

Action Required by July 22, 2025

Compute Plane VPC:
- For existing workspaces: If you have changed the security groups or network access lists of your workspaces, or you have a workspace in a customer-managed VPC, please ensure your egress rules allow these new ports.
- For new workspaces: Ensure the new ports are allowed in your security groups and network access lists.

3rd party firewall or Egress Proxy:
- If the traffic from your Databricks VPC is egressed through another appliance (e.g. a firewall), please ensure that these ports are also allow-listed in that appliance.

If you have any questions, please reach out to your Databricks account team or contact help@databricks.com

Databricks on GCP: Opening new ports for Databricks

GlorianaHM — Tue, 28 Oct 2025 05:35:23 GMT

The deadline for this action has been extended to November 10, 2025, from the original date of July 22, 2025.

To improve scalability and enable new features, the Databricks classic compute plane will start communicating to the Databricks control plane over 10 new ports: 3306 and 8443-8451. You may need to update your network configurations before we start using these ports on November 10, 2025.

Please ensure that the cloud administrator responsible for managing port configurations in your organization reviews the information below.

Action Required by November 10, 2025

Compute Plane VPC:
- For existing workspaces: If you have edited your VPC firewall, or you have a workspace in a customer-managed VPC, please ensure your egress rules allow these new ports.
- For new workspaces: Ensure the new ports are allowed in your VPC firewall.

Private Service Connect (PSC):
- If you are using PSC to connect the Databricks classic compute plane to the Databricks control plane, please reach out to your Databricks account team. They have been made aware of this change and will assist you in opening these ports.

3rd party firewall or Egress Proxy:
- If the traffic from your Databricks VPC is egressed through another appliance (e.g. a firewall), please ensure that these ports are also allow-listed in that appliance.

If you have any questions, please reach out to your Databricks account team or contact help@databricks.com

Databricks on AWS: Opening new ports for Databricks

GlorianaHM — Tue, 03 Jun 2025 17:58:48 GMT

Compute Plane VPC:
- For existing workspaces: If you have changed the security groups or network access lists of your workspaces, or you have a workspace in a customer-managed VPC, please ensure your egress rules allow these new ports.
- For new workspaces: Ensure the new ports are allowed in your security groups and network access lists.

3rd party firewall or Egress Proxy:
- If the traffic from your Databricks VPC is egressed through another appliance (e.g. a firewall), please ensure that these ports are also allow-listed in that appliance.

If you have any questions, please reach out to your Databricks account team or contact help@databricks.com

Network Configuration for Accessing Azure Key Vault from Enhanced Python UDFs in Unity Catalog

Shu_Li — Thu, 29 May 2025 15:33:47 GMT

Network configuration doesn’t have to be hard! In this article, I will walk you through different scenarios of network configuration for accessing Azure Key Vault(AKV) from Databricks.

Strengthening Data Privacy with Enhanced Python UDFs in Unity Catalog

Shu_Li — Tue, 27 May 2025 02:53:49 GMT

Unlock the next level of compliance and data privacy in your lakehouse with Databricks’ latest UC UDFs innovations. Discover how enhanced Unity Catalog (UC) user-defined functions (UDFs) empower you to implement custom data masking, tokenization, and encryption in this easy to follow, step-by-step guide

[Azure] Azure Databricks Upgrading to Standard SKU Public IP Addresses

GlorianaHM — Tue, 27 May 2025 02:53:05 GMT

Azure Databricks will gradually start adopting the Standard SKU for new public IPs on 11 April 2025

[Azure] Outbound public IP address ranges for the control plane will be updated May 20, 2025

GlorianaHM — Wed, 16 Jul 2025 04:28:01 GMT

To improve the security and zone support availability of the Azure Databricks control plane, we'll be updating the outbound public IP address range and associated Databricks service tags on May 20, 2025.

[Action required] [GCP] Databricks is updating the public IP range for our control plane services

AlexEsibov — Wed, 05 Mar 2025 05:12:47 GMT

This blog is to notify you that on January 20, 2025, Databricks will begin using new control plane service components on Google Cloud Platform (GCP). This will affect the egress and ingress IP addresses Databricks’ control plane uses. If you use a firewall or proxy appliance to restrict user access to Databricks control plane, and/or for controlling outbound access to your resources, you will need to update your access rules to include the new IP addresses. Otherwise, user access to the Databricks control plane, including the web app, may be blocked and Databricks’ control plane access to your resources may be blocked.

Please have the admin in your organization responsible for network security for the Databricks platform review the information below.

What’s changing

To support infrastructure improvements, Databricks is deploying new components for our control plane service. These changes will improve availability and routing infrastructure for our web app and control plane.

Beginning on January 20, 2025:

We will update the ingress Databricks control plane public IPs. These are the IP addresses listed for each region under “Control Plane IPs, including webapp”.
We will update the egress Databricks control plane public IPs. These are the IP addresses listed for each region under the Service “Control plane NAT”.

Action Required

If you have access rules on your firewall or gateway appliance to restrict access to or from the Databricks Control Plane, you will need to update the rules to include the new IP addresses for your region listed in the documentation.

If you have any questions or require additional support, please email support@databricks.com

Update your UC AWS IAM Roles to include self-assume capabilities by September 20th, 2024

JeffChen99 — Wed, 07 Aug 2024 09:40:13 GMT

TL;DR

Unity Catalog will begin prohibiting the creation of non-self-assuming IAM role storage credentials by September 20th, 2024, and will block the usage of existing non-self-assuming IAM role storage credentials by January 20th, 2025. This has the potential to break workloads and jobs that are still using these storage credentials at that time.

[Action required] [GCP] update outbound connectivity for classic workspaces

AlexEsibov — Wed, 16 Oct 2024 16:37:54 GMT

The following blog post details action required for workspaces that use workspace IP access lists. The communication will also be sent out by email to users who Databricks determines might be affected.

[Action required] [AWS] update outbound connectivity for classic workspaces

AlexEsibov — Wed, 16 Oct 2024 16:38:07 GMT

[Action required] [Azure] update outbound connectivity for classic workspaces

AlexEsibov — Wed, 16 Oct 2024 16:38:20 GMT

The following blog post details action required for workspaces that use both secure cluster connectivity and workspace IP access lists. The communication will also be sent out by email to users who Azure Databricks determines might be affected.

We are opening additional ports for Azure Databricks

AlexEsibov — Mon, 04 Nov 2024 17:19:11 GMT

IMPORTANT NOTE: we have delayed opening additional ports for new workspaces until January 31, 2025. For existing workspaces, Network Security Groups (NSGs) have been updated. If you received a communication that updates for your existing workspaces failed, please read on.

If updates to your existing workspaces failed, you must manually modify your NSGs to allow outbound traffic to the Azure Databricks control plane over 10 new ports: 3306, 8443-8451.

If your NSG is in a locked Azure resource group or subscription, please remove the lock before performing the update. Also, ensure that you are not denying any resource changes to your NSG with an Azure Policy before performing the update.

Please refer to the NSG rules here as a reference: Deploy Azure Databricks in your Azure virtual network (VNet injection)

Note that if you have multiple workspaces using a single NSG, we may have detected a false positive workspace update failure. If the outbound rule to destination "AzureDatabricks (service tag)" contains ports 8443-8451, the update was successfully applied.

----------------------

Original Communication below

Background: To improve scalability and enable new features, the Azure Databricks control plane will communicate with the classic compute plane over 10 new ports.

We’ll be updating your existing network security groups (NSGs) used for the public and private subnets of your VNet injection workspaces with an additional rule to allow outbound traffic to the Azure Databricks control plane over 10 new ports: 3306 and 8443–8451. Existing security rules already allow outbound traffic over port 443 for VNet injection workspaces and, for back-end private link workspaces, port 6666.

Action:

We'll open the new ports on your behalf, and most customers won’t need to take any action. However, if you have existing rules that disallow these ports, you may need to update these rules before this date.

If you have existing VNet injection workspaces, you’ll need to ensure that no NSGs on your public and private subnets block the new ports.
If you use back-end private link, ensure that no NSGs on your private endpoints block these ports.
Ensure no firewall rules block these ports.
Note that the NSG rules that Azure Databricks auto-provisions have been replaced with equivalent rules with custom names, duplicate rules may be created. This won't affect functionality.

If there is a rule blocking these ports at the time Databricks pushes this update, the update will fail. However, in this event, workspaces will continue to function properly for existing features. You will know that the update failed if the NSG does not have the new ports explicitly included after date of the change.

Timeline:

For existing workspaces, we will open the new ports on June 3, 2024
For new workspaces, we will open the new ports on July 1, 2024. Ensure any workspace creation automation is also updated by then.

In both cases, please allow up to 2 weeks for the change to rollout to all regions.

If you are not the admin responsible for network connectivity to Azure Databricks, please forward this email to that person. If you have questions, get answers from community experts in Microsoft Q&A. If you have a support plan and need technical help, please create a support request.

[Action required] Azure Databricks is updating the public IP range for its control plane services

AlexEsibov — Wed, 27 Mar 2024 20:41:42 GMT

Starting on 30 May 2024, Azure Databricks will begin using new control plane service components. This will affect the egress and ingress IP addresses Azure Databricks’ control plane uses. If you use a firewall or proxy appliance to restrict user access to Azure Databricks control plane, and/or for controlling outbound access to your resources, you will need to update your access rules to include the new IP addresses. Otherwise, user access to the Azure Databricks control plane, including the web app, may be blocked and Azure Databricks’ control plane access to your resources may be blocked.

Please have the admin in your organization responsible for network security for the Azure Databricks platform review the information below.

What's changing

To support infrastructure improvements, Azure Databricks is deploying new components for our control plane service. These changes will improve multi-zonal availability and routing infrastructure for our web app and control plane.

Beginning on 30 May 2024, this means that:

We will update the ingress Azure Databricks Control Plane Public IPs and associated Azure Service Tags. These are the IP addresses listed for each region under “Control Plane IPs, including webapp”.
We will update the egress Azure Databricks Control Plane Public IPs and associated Azure Service Tags. These are the IP addresses listed for each region under the Service “Control Plane NAT”.

Action required

If you have access rules on your firewall or gateway appliance to restrict access to Azure Databricks Control Plane ingress IPs, you will need to update the rules to include the new IP addresses for your region. See link in resources.
- Note that if you use Azure Service tags in your Azure Firewall, we will update the existing service tags to include the new IP addresses and no action is required with regards to #1. If you are using Azure Firewall but not leveraging service tags today, we recommend you migrate to using service tags if possible. See link in resources.
If you have access rules in any resource firewall or proxy that restrict access from Azure Databricks Control Plane egress IPs, you will need to update the rules to include the new “Control Plane NAT” IP addresses. See link in resources.

Resources

The list of all Azure Databricks control plane ingress IP addresses that must be included in your user/service access policies can be found here, under “Control Plane IPs, including webapp” for your region(s)
The list of all Azure Databricks control plane egress IP addresses that must be included in your resource firewall policies can be found here, under “Control Plane NAT” for your region(s)
More information on using Azure Service Tags can be found here

If you have any questions or require additional support, please comment below or open a support ticket with Azure Databricks.

Thank you,

Azure Databricks

DigiCert is updating its root CA certificate

AlexEsibov — Sat, 16 Mar 2024 14:47:43 GMT

Databricks uses DigiCert as its SSL certificate authority, to establish trust. DigiCert is making a change to its root and intermediate CA certificates (detailed information below). This change will impact Databricks, starting on May 15, 2024. If you use one or more supported browsers or any other client that trusts DigiCert’s new root and intermediate CA certificates, no action is required. For all other clients, you should ensure that you establish trust with the new DigiCert root and intermediate CA certificates. Please see below for information on how to install and manage certificates, how to test that the cert has been installed correctly, and for more information on the DigiCert change.

Testing the change

Once your client trusts Digicert’s new root and intermediate CA certificates, you will be able to connect to this endpoint, with a secure TLS connection: https://nvirginia.staging.cloud.databricks.com/login.html

Other resources

Detailed information about the DigiCert change
For reference only, examples of how to install and manage client certificates

Thank you

Product Platform Updates articles

Azure Databricks: Required change in resource firewalls allowlisting Databricks serverless compute

IP CIDR For Azure Databricks Hive Metastore

What's new in Databricks - August 2025

Azure Databricks — Upgrade Managed Workspace to VNet Injected Workspace

Managed Workspace in Azure Databricks

Prerequisites

Limitations and Other Considerations

Convert a managed workspace to a VNet Injected workspace

Step-1 Create Network Security Group

Step-2 Create Virtual Network

Step-3 Upgrade the Workspace to VNet Injection

Conclusion

What's new in Databricks: July 2025

ai_parse_document is now available in the following regions (AWS):

us-west-2/ us-east-1/ us-east-2 /ap-northeast-1 /ap-northeast-2 /ap-south-1 /ap-southeast-1

ap-southeast-2/ca-central-1/eu-central-1 /eu-west-1 /eu-west-2 /sa-east-1

Databricks on AWS: Opening new ports for Databricks

Databricks on GCP: Opening new ports for Databricks

Databricks on AWS: Opening new ports for Databricks

Network Configuration for Accessing Azure Key Vault from Enhanced Python UDFs in Unity Catalog

Strengthening Data Privacy with Enhanced Python UDFs in Unity Catalog

[Azure] Azure Databricks Upgrading to Standard SKU Public IP Addresses

[Azure] Outbound public IP address ranges for the control plane will be updated May 20, 2025

[Action required] [GCP] Databricks is updating the public IP range for our control plane services

What’s changing

Action Required

Update your UC AWS IAM Roles to include self-assume capabilities by September 20th, 2024

[Action required] [GCP] update outbound connectivity for classic workspaces

[Action required] [AWS] update outbound connectivity for classic workspaces

[Action required] [Azure] update outbound connectivity for classic workspaces

We are opening additional ports for Azure Databricks

[Action required] Azure Databricks is updating the public IP range for its control plane services

What's changing

Action required

Resources

DigiCert is updating its root CA certificate

`ai_parse_document` is now available in the following regions (AWS):

`us-west-2/ us-east-1/ us-east-2 /ap-northeast-1 /ap-northeast-2 /ap-south-1 /ap-southeast-1`

`ap-southeast-2/ca-central-1/eu-central-1 /eu-west-1 /eu-west-2 /sa-east-1`