a week ago
Hi all,
A full refresh of one Auto Loader streaming table in a serverless #Lakeflow (Declarative) pipeline has become about 4x slower (all to do with a single streaming table), and I'd like help understanding why. Nothing on our side changed. The only difference I can find between a fast run and a slow run is the pipeline runtime build and the fact that we turned on managed file events, which have been improving file listing. I can see that the compute isn't scaling out as much as it was previously. It seems only 1-2 concurrent tasks are executing, rather than 4+ previously.
SETUP
- Compute: serverless, triggered pipeline, CURRENT channel (Azure)
- Source: Auto Loader with cloudFiles.format = json and cloudFiles.schemaEvolutionMode = addNewColumns
- Files: each Kafka message lands as its own JSON file, about 20-45 KB each
- Rebuild size: about 190k files in our test environment, about 1.3M files in pre-prod
THE COMPARISON
I ran a full refresh of only this table in the same test environment. Each time the files, code, schema, reader options and table properties were the same, managed file events were off, and nothing else was running in the pipeline.
Older run (October)
- Runtime image: dlt:18.3.7-...-aarch64-photon-...-dp-20261005
- Files: about 191k
- Ingest time after the first batch: about 3 minutes
- Throughput: about 1,000 files/s
- Micro-batches running at once: always 4
Newer run (November)
- Runtime image: dlt:18.3.10-...-photon-...-dp-20261106
- Files: about 194k
- Ingest time after the first batch: about 12 minutes
- Throughput: about 260 files/s
- Micro-batches running at once: mostly 1-2
(I judged how many micro-batches ran at once from when they finish in the event log: groups of batches finishing within about 1 second of each other.)
With maxFilesPerTrigger removed, so Auto Loader sizes the batches itself (15k-90k files per batch), the newer runtime went back to four batches at once. That run still took about 9 minutes, about 370 files/s, which is about a third of the old speed.
QUERY PROFILE (ONE 10K-FILE BATCH)
- Share of scan time spent waiting on I/O: about 98-99%
- Average storage request time: about 14-16 ms
- Storage retries / throttling: 0
- Task time per file: about 21-24 ms (similar on both runtimes)
- Scan tasks per batch: 80
- Prefetch, number of files: 0
Total scan time divided by wall time gives:
- Older runtime: about 21-24 tasks running at once
- Newer runtime: about 7-8 tasks running at once across the whole pipeline
My reading is that the work per file hasn't changed, but the newer runtime gets about a third of the compute and doesn't scale up. The backlog stays above 100k files with plenty of queued tasks, yet throughput stays flat for the whole run. In pre-prod the same rebuild ran at about 300 files/s for over an hour and never sped up.
WHAT I'VE RULED OUT
- Managed file events: same result with them on or off
- Other flows competing: I refreshed only this one table
- Data volume: both runs read about the same number of files
- maxFilesPerTrigger: 10k fixed or removed. Removing it helped about 40%, but the ceiling stays.
- Storage latency: about the same per request in both runs, with no retries
Can someone please help me figure this out? because on production workloads, this is becoming a real problem.
Thanks in advance. I'm happy to share more metrics if they'd help.
a week ago
Your metrics already narrow this down significantly. Per-file processing time (~21–24 ms), storage latency, retries, and tasks per batch are all unchanged, yet concurrent running tasks dropped from ~21–24 down to ~7–8 while the backlog stayed above 100k. This makes an Auto Loader read-path regression very unlikely. Your stream is being starved of task slots; the real question is why serverless stopped allocating them.
Serverless pipelines manage compute sizing automatically by layering vertical autoscaling on top of Databricks Enhanced Autoscaling, which includes CPU-driven scaling logic that adjusts resources based on CPU utilization.
Because your scan spends 98–99% of its time waiting on I/O, the CPUs look nearly idle even with a massive task queue. Scaling logic that weighs CPU utilization heavily will interpret this as an oversized cluster and hold back capacity, matching the flat throughput ceiling you're seeing across an hour-long rebuild. While CPU-based scaling predates your October run, a change in the November runtime build likely altered how aggressively the autoscaler reacts to low-CPU, high-I/O workloads.
This also explains the drop in batch concurrency. Serverless pipelines use stream pipelining to run microbatches concurrently rather than sequentially. Fewer available task slots mean fewer overlapping batches, accounting for the drop from 4 concurrent batches down to 1–2.
Since serverless pipelines don't populate cluster_resources events in the event log, start with these checks:
Verify the exact runtime builds: Pull the image_key field from the runtime_details event for both update IDs to confirm the exact Databricks Runtime versions used.
Compare backlog vs. throughput: Plot flow_progress backlog metrics over time for both runs. Flat throughput alongside a steady or growing backlog points directly to a compute capacity ceiling rather than a source bottleneck.
Inspect the Query Profile: Your scan-time-to-wall-time ratio is the right metric here. Use the Query Profile to verify the drop in effective parallelism.
Run a Classic Compute baseline: Run the same full refresh once on classic compute using a fixed worker count sized close to your October footprint (~24 cores). Classic compute populates cluster_resources, allowing you to inspect slot allocation and queue depth directly:
SELECT timestamp, details:cluster_resources.num_executors, details:cluster_resources.num_task_slots, details:cluster_resources.avg_task_slot_utilization, details:cluster_resources.avg_num_queued_tasks FROM event_log(TABLE(<catalog>.<schema>.<your_table>)) WHERE event_type = 'cluster_resources' AND origin.update_id = '<classic_update_id>' ORDER BY timestamp;
If classic compute gets you back to ~1,000 files/sec on the same runtime build, you have proof that the bottleneck is serverless capacity allocation rather than the read path.
Instance architecture: Your October image string references aarch64, whereas the November string you shared does not. Unless that was trimmed when pasting, a shift in underlying instance family could change the task-slots-per-node ratio.
Performance mode: For triggered pipelines, check the pipeline scheduler settings and ensure Performance optimized mode is enabled. If disabled, the pipeline defaults to standard performance mode, which throttles compute usage to save DBUs.
Prefetch = 0: Since this setting is identical across both runs, it isn't the cause of the regression; it just explains why throughput on tiny files is so sensitive to total task slot count.
Test the PREVIEW channel: Run the pipeline against the preview channel to see if an upcoming runtime update already resolves the scaling behavior.
Test bytes-based batching: Try setting cloudFiles.maxBytesPerTrigger. When combined with maxFilesPerTrigger, Auto Loader triggers when whichever limit is reached first. Since removing the file cap already gave you a ~40% boost, sizing microbatches by byte volume is worth testing.
Address small files upstream: Storing one Kafka message per 20–45 KB file is the underlying bottleneck. If feasible, ingest from Kafka directly into your bronze table or compact/batch messages into larger files at landing so full refreshes aren't so heavily dependent on wide horizontal scaling.
If the classic compute test confirms the bottleneck, open a ticket with Databricks Support and include both update IDs, both image_key values, and your classic-vs-serverless metrics. A 3–4x throughput drop on identical inputs following a runtime update, with zero code or config changes on your end, gives them clear grounds to inspect the internal autoscaler logs.
a week ago
Hello @rjmcbride, I took a look at both internal and external documentation and here is what I found.
This is one of the cleaner performance writeups I’ve seen on here. Same files, same code, same options, one variable changed, and both runs measured the same way. That makes everyone else more useful to you.
@ashishshivatare’s read is the one I’d start from. Per-file cost, storage latency, and tasks per batch are unchanged, so the read path isn’t the problem. Effective task slots fell from about 21 to 24 down to about 7 to 8, and that alone accounts for nearly all of the slowdown. The question is why serverless stopped handing out slots, and the timing points at the move from dlt:18.3.7...dp-20261005 to dlt:18.3.10...dp-20261106.
Two mechanisms are in play, and they watch different signals. Enhanced autoscaling (the horizontal piece) is documented as using task slot utilization and task queue size. Not CPU. With 100k+ files in the backlog and tasks queued all run, the documented inputs say “scale out,” so the fact that it didn’t is itself the anomaly, and it’s the sentence I’d put at the top of a Support ticket.
CPU comes in through vertical autoscaling. The March 2026 release notes say serverless pipelines gained CPU-based vertical autoscaling that adjusts resources based on actual CPU utilization. Your scan spends 98 to 99 percent of its time waiting on storage, so the CPUs look idle even with a mountain of queued work. That’s a plausible path to smaller nodes and fewer slots. It’s a hypothesis, not a diagnosis. The feature predates your October run, and I don’t know what changed in the November build.
As for “randomly”: channel upgrades roll out region by region, so two workspaces, or the same workspace a week apart, can be on different builds. Line up image_key against throughput for every update you’ve compared and I suspect the pattern holds.
dbr_version and image_key from the runtime_details event for a fast and a slow update. Keep the full string. Your October image says aarch64 and your November one doesn’t; if that wasn’t trimmed when pasting, you have an ARM to x86 instance change in the mix, which changes slots per node.backlog_files and backlog_bytes from flow_progress METRICS events for both updates. Flat throughput over a steady backlog is the capacity-ceiling signature. The system.lakeflow_pipeline_events_preview.pipeline_events table (Beta) lets you compare updates in one query.cluster_resources and autoscale events are only emitted for classic pipelines, so this is the only way to see slot allocation and queue depth directly. If classic on the same DBR gets you back near 1,000 files/s, it’s serverless capacity, not your code or storage.With the file cap removed, the pipeline sized batches at 15k to 90k files itself and concurrency came back, so I’d leave the cap off. cloudFiles.maxBytesPerTrigger is a reasonable test too; the file limit is hard, the byte limit is soft, and with both set Auto Loader stops at whichever it reaches first.
The real bottleneck is one Kafka message per 20 to 45 KB file. Every record costs a 14 to 16 ms round trip, and the only knob inside the pipeline is concurrency, which on serverless you don’t control. Reading Kafka directly into the bronze streaming table removes the file hop. If that’s not an option, batching messages into larger files at landing makes every future refresh less sensitive to changes like this one.
Open the ticket now rather than after the classic test. A 3 to 4x drop on identical inputs that tracks a runtime build is exactly what they need, and the public release notes (through September 2026) don’t mention anything that explains it. Include:
image_key valuesPlease post back what Support finds. Others running small-file ingestion on serverless are going to run into this, and your thread is the best writeup of the symptom I’ve seen.
References:
runtime_details, flow_progress, cluster_resources😞 https://learn.microsoft.com/azure/databricks/ldp/monitor-event-log-schemaRegards,
Louis
a week ago
a week ago
@saurabh18cs , That is classic compute. Serverless does not allow the configuration of max workers.