Databricks has rolled out Runtime 18.1 (Beta), and itโs packed with meaningful enhancements across streaming, Delta Lake, SQL, geospatial, performance, and Spark 4.1.0 improvements. This release builds on 18.0 and introduces new capabilities that make pipelines faster, smarter, and more reliable. Hereโs a breakdown of whatโs new and why it matters.
Key New Features & Improvements
๐Auto Loader Enhancements
Auto Loader now uses file events by default when available, reducing directory listing costs and improving latency. You can still override behaviour using useIncrementalListing, useNotifications, or disable file events with useManagedFileEvents = false.
๐Delta Lake & Unity Catalog Improvements
Optimized Writes for CRTAS
Partitioned Unity Catalog tables created via CREATE OR REPLACE TABLE AS SELECT now automatically use optimized writes for fewer, larger files.
๐Schema Evolution with INSERT
New WITH SCHEMA EVOLUTION clause allows automatic schema evolution during INSERT INTO, INSERT OVERWRITE, and INSERT INTO โฆ REPLACE.
Handles new columns, widened types, and preserves NULL struct values even when field order differs.
๐Delta Sharing
Now supports multiโstatement transactions for shared tables using preโsigned URLs or cloud tokens.
๐SQL & Scripting Enhancements
New SQL Functions
parse_timestamp โ photonized for fast multiโpattern timestamp parsing.
Approximate topโk sketch functions (approx_top_k_accumulate, approx_top_k_combine, approx_top_k_estimate).
Tuple sketch functions for distinct counting and keyโsummary aggregation.
๐SQL Cursor Support
Compound SQL statements now support DECLARE CURSOR, OPEN, FETCH, and CLOSE for rowโbyโrow processing.
๐Behavioural Changes
FILTER clause now works with MEASURE aggregate functions.
Timestamp partitions now use Spark session timezone instead of JVM timezone.
DESCRIBE FLOW is now a reserved keyword.
๐ Streaming Improvements
Automatic streaming type widening for Delta tables.
New configs allow stricter control if needed.
๐ Geospatial Performance Boost
Geospatial Boolean set operations now use a new, faster implementation (with minor precision differences beyond 15 decimal places).
๐ DataFrame & Compute Enhancements
DataFrame checkpoints now support Unity Catalog volume paths.
.cache() no longer reโruns SQL commands like SHOW TABLES.
๐ Cloud & External System Improvements
DATETIMEOFFSET type support for Azure Synapse.
Google BigQuery table descriptions now appear as table comments.
๐ Apache Spark 4.1.0 Included
Databricks Runtime 18.1 ships with Apache Spark 4.1.0, bringing:
Major performance fixes
Improved pandas interoperability
New geospatial type support
Arrow & Pandas UDF improvements
Streaming enhancements
Stability and errorโhandling improvements
If you're building modern data platforms, experimenting with LLMs, or optimizing production pipelines, this runtime is absolutely worth exploring.
#Databricks #RuntimeLatest #Beta #DatabricksMVP
