🚀 Databricks Runtime 18.1 (Beta) — A Big Leap Forward for Data & AI Teams

Abiola-David — Wed, 25 Feb 2026 22:20:03 GMT

Databricks has rolled out Runtime 18.1 (Beta), and it’s packed with meaningful enhancements across streaming, Delta Lake, SQL, geospatial, performance, and Spark 4.1.0 improvements. This release builds on 18.0 and introduces new capabilities that make pipelines faster, smarter, and more reliable. Here’s a breakdown of what’s new and why it matters.

Key New Features & Improvements

📊Auto Loader Enhancements
Auto Loader now uses file events by default when available, reducing directory listing costs and improving latency. You can still override behaviour using useIncrementalListing, useNotifications, or disable file events with useManagedFileEvents = false.

📊Delta Lake & Unity Catalog Improvements
Optimized Writes for CRTAS
Partitioned Unity Catalog tables created via CREATE OR REPLACE TABLE AS SELECT now automatically use optimized writes for fewer, larger files.

📊Schema Evolution with INSERT
New WITH SCHEMA EVOLUTION clause allows automatic schema evolution during INSERT INTO, INSERT OVERWRITE, and INSERT INTO … REPLACE.
Handles new columns, widened types, and preserves NULL struct values even when field order differs.

📊Delta Sharing
Now supports multi‑statement transactions for shared tables using pre‑signed URLs or cloud tokens.

📊SQL & Scripting Enhancements
New SQL Functions
parse_timestamp — photonized for fast multi‑pattern timestamp parsing.
Approximate top‑k sketch functions (approx_top_k_accumulate, approx_top_k_combine, approx_top_k_estimate).
Tuple sketch functions for distinct counting and key‑summary aggregation.

📊SQL Cursor Support
Compound SQL statements now support DECLARE CURSOR, OPEN, FETCH, and CLOSE for row‑by‑row processing.

📊Behavioural Changes
FILTER clause now works with MEASURE aggregate functions.
Timestamp partitions now use Spark session timezone instead of JVM timezone.
DESCRIBE FLOW is now a reserved keyword.

📊 Streaming Improvements
Automatic streaming type widening for Delta tables.
New configs allow stricter control if needed.

📊 Geospatial Performance Boost
Geospatial Boolean set operations now use a new, faster implementation (with minor precision differences beyond 15 decimal places).

📊 DataFrame & Compute Enhancements
DataFrame checkpoints now support Unity Catalog volume paths.
.cache() no longer re‑runs SQL commands like SHOW TABLES.

📊 Cloud & External System Improvements
DATETIMEOFFSET type support for Azure Synapse.
Google BigQuery table descriptions now appear as table comments.

📊 Apache Spark 4.1.0 Included
Databricks Runtime 18.1 ships with Apache Spark 4.1.0, bringing:
Major performance fixes
Improved pandas interoperability
New geospatial type support
Arrow & Pandas UDF improvements
Streaming enhancements
Stability and error‑handling improvements

If you're building modern data platforms, experimenting with LLMs, or optimizing production pipelines, this runtime is absolutely worth exploring.

#Databricks #RuntimeLatest #Beta #DatabricksMVP

topic 🚀 Databricks Runtime 18.1 (Beta) — A Big Leap Forward for Data & AI Teams in MVP Articles

🚀 Databricks Runtime 18.1 (Beta) — A Big Leap Forward for Data & AI Teams