How to force Databricks Serverless to process 1 TB (40 Billion Rows) in 14 Minutes

ShamenParis — Fri, 18 Sep 2026 17:17:12 GMT

Databricks Serverless is incredible for abstracting infrastructure, but because it scales based on the volume of pending tasks, relying on default configurations for massive data ingestion can leave performance on the table.

I recently ran an experiment to ingest 91 massive Parquet files (6–8 GB each) totaling 1.04 TB. Using standard 128MB partitions, this only generates a few thousand tasks—not enough to trigger the Serverless engine’s maximum horizontal scale-out.

Here is the one-line configuration hack to change that: spark.conf.set("spark.sql.files.maxPartitionBytes", "33554432")

By dropping the max partition size to 32MB, we force Spark to aggressively split those massive files into roughly 23,000 micro-tasks. When the Serverless auto-scaler detects a queue that large, it instantly provisions a massive fleet of Photon-enabled workers, pushing read throughput over 1.2 GB/s.

The Results & The "Gotcha" While the engine chewed through 40.1 billion rows in just 14 minutes and 5 seconds, it actually read 3x more rows than it wrote (13.3 billion).

Why? Cluster-on-write.

Because the target Delta table was created with CLUSTER BY (sales_date, warehouse_name), the Serverless engine had to perform a massive in-memory shuffle of all 40 billion rows before writing. Crunching a 1 TB shuffle in 14 minutes with zero infrastructure management is a huge win for Serverless compute, but it technically missed my sub-10-minute SLA for this pipeline.

The Architecture Fix for Maximum Speed: If you need absolute maximum raw I/O throughput:

Decouple the clustering: Create the initial Delta table without clustering keys.
Ingest raw: Execute the write phase as a pure, map-only I/O operation (no shuffle). With the 32MB partition trick, this 1 TB load drops into single-digit minutes.
Defer the heavy lifting: Run ALTER TABLE ... CLUSTER BY after the write. Serverless Predictive Optimization will seamlessly sort the underlying files asynchronously in the background.

Check my LinkedIn post: https://lnkd.in/p/e3xUqq5b

Has anyone else been experimenting with partition sizing to manipulate Serverless auto-scaling? Let me know your findings below. 👇

#Databricks #DataEngineering #ApacheSpark #Serverless #BigData #PerformanceTuning #DeltaLake

topic How to force Databricks Serverless to process 1 TB (40 Billion Rows) in 14 Minutes in Community Articles

How to force Databricks Serverless to process 1 TB (40 Billion Rows) in 14 Minutes