cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Serverless declarative pipeline ingestion from ADLS is running x4 SLOWER randomly

rjmcbride
New Contributor

 

Hi all,

A full refresh of one Auto Loader streaming table in a serverless #Lakeflow (Declarative) pipeline has become about 4x slower (all to do with a single streaming table), and I'd like help understanding why. Nothing on our side changed. The only difference I can find between a fast run and a slow run is the pipeline runtime build and the fact that we turned on managed file events, which have been improving file listing. I can see that the compute isn't scaling out as much as it was previously. It seems only 1-2 concurrent tasks are executing, rather than 4+ previously.


SETUP

- Compute: serverless, triggered pipeline, CURRENT channel (Azure)
- Source: Auto Loader with cloudFiles.format = json and cloudFiles.schemaEvolutionMode = addNewColumns
- Files: each Kafka message lands as its own JSON file, about 20-45 KB each
- Rebuild size: about 190k files in our test environment, about 1.3M files in pre-prod


THE COMPARISON

I ran a full refresh of only this table in the same test environment. Each time the files, code, schema, reader options and table properties were the same, managed file events were off, and nothing else was running in the pipeline.

Older run (October)
- Runtime image: dlt:18.3.7-...-aarch64-photon-...-dp-20261005
- Files: about 191k
- Ingest time after the first batch: about 3 minutes
- Throughput: about 1,000 files/s
- Micro-batches running at once: always 4

Newer run (November)
- Runtime image: dlt:18.3.10-...-photon-...-dp-20261106
- Files: about 194k
- Ingest time after the first batch: about 12 minutes
- Throughput: about 260 files/s
- Micro-batches running at once: mostly 1-2

(I judged how many micro-batches ran at once from when they finish in the event log: groups of batches finishing within about 1 second of each other.)

With maxFilesPerTrigger removed, so Auto Loader sizes the batches itself (15k-90k files per batch), the newer runtime went back to four batches at once. That run still took about 9 minutes, about 370 files/s, which is about a third of the old speed.


QUERY PROFILE (ONE 10K-FILE BATCH)

- Share of scan time spent waiting on I/O: about 98-99%
- Average storage request time: about 14-16 ms
- Storage retries / throttling: 0
- Task time per file: about 21-24 ms (similar on both runtimes)
- Scan tasks per batch: 80
- Prefetch, number of files: 0

Total scan time divided by wall time gives:
- Older runtime: about 21-24 tasks running at once
- Newer runtime: about 7-8 tasks running at once across the whole pipeline

My reading is that the work per file hasn't changed, but the newer runtime gets about a third of the compute and doesn't scale up. The backlog stays above 100k files with plenty of queued tasks, yet throughput stays flat for the whole run. In pre-prod the same rebuild ran at about 300 files/s for over an hour and never sped up.


WHAT I'VE RULED OUT

- Managed file events: same result with them on or off
- Other flows competing: I refreshed only this one table
- Data volume: both runs read about the same number of files
- maxFilesPerTrigger: 10k fixed or removed. Removing it helped about 40%, but the ceiling stays.
- Storage latency: about the same per request in both runs, with no retries

Can someone please help me figure this out? because on production workloads, this is becoming a real problem.

Thanks in advance. I'm happy to share more metrics if they'd help.

4 REPLIES 4

ashishshivatare
Databricks Partner

Your metrics already narrow this down significantly. Per-file processing time (~21โ€“24 ms), storage latency, retries, and tasks per batch are all unchanged, yet concurrent running tasks dropped from ~21โ€“24 down to ~7โ€“8 while the backlog stayed above 100k. This makes an Auto Loader read-path regression very unlikely. Your stream is being starved of task slots; the real question is why serverless stopped allocating them.

Likely Cause: CPU-Driven Autoscaling on an I/O-Bound Workload

Serverless pipelines manage compute sizing automatically by layering vertical autoscaling on top of Databricks Enhanced Autoscaling, which includes CPU-driven scaling logic that adjusts resources based on CPU utilization.

Because your scan spends 98โ€“99% of its time waiting on I/O, the CPUs look nearly idle even with a massive task queue. Scaling logic that weighs CPU utilization heavily will interpret this as an oversized cluster and hold back capacity, matching the flat throughput ceiling you're seeing across an hour-long rebuild. While CPU-based scaling predates your October run, a change in the November runtime build likely altered how aggressively the autoscaler reacts to low-CPU, high-I/O workloads.

This also explains the drop in batch concurrency. Serverless pipelines use stream pipelining to run microbatches concurrently rather than sequentially. Fewer available task slots mean fewer overlapping batches, accounting for the drop from 4 concurrent batches down to 1โ€“2.

How to Confirm It

Since serverless pipelines don't populate cluster_resources events in the event log, start with these checks:

  1. Verify the exact runtime builds: Pull the image_key field from the runtime_details event for both update IDs to confirm the exact Databricks Runtime versions used.

  2. Compare backlog vs. throughput: Plot flow_progress backlog metrics over time for both runs. Flat throughput alongside a steady or growing backlog points directly to a compute capacity ceiling rather than a source bottleneck.

  3. Inspect the Query Profile: Your scan-time-to-wall-time ratio is the right metric here. Use the Query Profile to verify the drop in effective parallelism.

  4. Run a Classic Compute baseline: Run the same full refresh once on classic compute using a fixed worker count sized close to your October footprint (~24 cores). Classic compute populates cluster_resources, allowing you to inspect slot allocation and queue depth directly:

SQL
 
SELECT
  timestamp,
  details:cluster_resources.num_executors,
  details:cluster_resources.num_task_slots,
  details:cluster_resources.avg_task_slot_utilization,
  details:cluster_resources.avg_num_queued_tasks
FROM event_log(TABLE(<catalog>.<schema>.<your_table>))
WHERE event_type = 'cluster_resources'
  AND origin.update_id = '<classic_update_id>'
ORDER BY timestamp;

If classic compute gets you back to ~1,000 files/sec on the same runtime build, you have proof that the bottleneck is serverless capacity allocation rather than the read path.

Other Variables Worth Checking

  • Instance architecture: Your October image string references aarch64, whereas the November string you shared does not. Unless that was trimmed when pasting, a shift in underlying instance family could change the task-slots-per-node ratio.

  • Performance mode: For triggered pipelines, check the pipeline scheduler settings and ensure Performance optimized mode is enabled. If disabled, the pipeline defaults to standard performance mode, which throttles compute usage to save DBUs.

  • Prefetch = 0: Since this setting is identical across both runs, it isn't the cause of the regression; it just explains why throughput on tiny files is so sensitive to total task slot count.

Workarounds in the Meantime

  • Test the PREVIEW channel: Run the pipeline against the preview channel to see if an upcoming runtime update already resolves the scaling behavior.

  • Test bytes-based batching: Try setting cloudFiles.maxBytesPerTrigger. When combined with maxFilesPerTrigger, Auto Loader triggers when whichever limit is reached first. Since removing the file cap already gave you a ~40% boost, sizing microbatches by byte volume is worth testing.

  • Address small files upstream: Storing one Kafka message per 20โ€“45 KB file is the underlying bottleneck. If feasible, ingest from Kafka directly into your bronze table or compact/batch messages into larger files at landing so full refreshes aren't so heavily dependent on wide horizontal scaling.

Opening a Support Ticket

If the classic compute test confirms the bottleneck, open a ticket with Databricks Support and include both update IDs, both image_key values, and your classic-vs-serverless metrics. A 3โ€“4x throughput drop on identical inputs following a runtime update, with zero code or config changes on your end, gives them clear grounds to inspect the internal autoscaler logs.

Louis_Frolio
Databricks Employee
Databricks Employee

Hello @rjmcbride, I took a look at both internal and external documentation and here is what I found.

This is one of the cleaner performance writeups Iโ€™ve seen on here. Same files, same code, same options, one variable changed, and both runs measured the same way. That makes everyone else more useful to you.

@ashishshivatareโ€™s read is the one Iโ€™d start from. Per-file cost, storage latency, and tasks per batch are unchanged, so the read path isnโ€™t the problem. Effective task slots fell from about 21 to 24 down to about 7 to 8, and that alone accounts for nearly all of the slowdown. The question is why serverless stopped handing out slots, and the timing points at the move from dlt:18.3.7...dp-20261005 to dlt:18.3.10...dp-20261106.

What the docs say about scaling

Two mechanisms are in play, and they watch different signals. Enhanced autoscaling (the horizontal piece) is documented as using task slot utilization and task queue size. Not CPU. With 100k+ files in the backlog and tasks queued all run, the documented inputs say โ€œscale out,โ€ so the fact that it didnโ€™t is itself the anomaly, and itโ€™s the sentence Iโ€™d put at the top of a Support ticket.

CPU comes in through vertical autoscaling. The March 2026 release notes say serverless pipelines gained CPU-based vertical autoscaling that adjusts resources based on actual CPU utilization. Your scan spends 98 to 99 percent of its time waiting on storage, so the CPUs look idle even with a mountain of queued work. Thatโ€™s a plausible path to smaller nodes and fewer slots. Itโ€™s a hypothesis, not a diagnosis. The feature predates your October run, and I donโ€™t know what changed in the November build.

As for โ€œrandomlyโ€: channel upgrades roll out region by region, so two workspaces, or the same workspace a week apart, can be on different builds. Line up image_key against throughput for every update youโ€™ve compared and I suspect the pattern holds.

Tests, in the order Iโ€™d run them

  1. Pull dbr_version and image_key from the runtime_details event for a fast and a slow update. Keep the full string. Your October image says aarch64 and your November one doesnโ€™t; if that wasnโ€™t trimmed when pasting, you have an ARM to x86 instance change in the mix, which changes slots per node.
  2. Confirm Performance optimized is on for the production schedule. The docs frame standard mode as slower startup and fewer DBUs reflecting lower compute usage. Iโ€™m not certain how a manually started full refresh is treated, so check your test runs both ways.
  3. Plot backlog_files and backlog_bytes from flow_progress METRICS events for both updates. Flat throughput over a steady backlog is the capacity-ceiling signature. The system.lakeflow_pipeline_events_preview.pipeline_events table (Beta) lets you compare updates in one query.
  4. Run the full refresh once on classic compute with a fixed worker count near your October footprint. cluster_resources and autoscale events are only emitted for classic pipelines, so this is the only way to see slot allocation and queue depth directly. If classic on the same DBR gets you back near 1,000 files/s, itโ€™s serverless capacity, not your code or storage.
  5. Try the Preview channel on a staging copy. Preview is on DBR 19 while Current is on DBR 18. If itโ€™s fast, the fix may already be on its way. Keep production on Current.

Batch sizing and the file shape

With the file cap removed, the pipeline sized batches at 15k to 90k files itself and concurrency came back, so Iโ€™d leave the cap off. cloudFiles.maxBytesPerTrigger is a reasonable test too; the file limit is hard, the byte limit is soft, and with both set Auto Loader stops at whichever it reaches first.

The real bottleneck is one Kafka message per 20 to 45 KB file. Every record costs a 14 to 16 ms round trip, and the only knob inside the pipeline is concurrency, which on serverless you donโ€™t control. Reading Kafka directly into the bronze streaming table removes the file hop. If thatโ€™s not an option, batching messages into larger files at landing makes every future refresh less sensitive to changes like this one.

Support

Open the ticket now rather than after the classic test. A 3 to 4x drop on identical inputs that tracks a runtime build is exactly what they need, and the public release notes (through September 2026) donโ€™t mention anything that explains it. Include:

  • both update IDs and both image_key values
  • the two query profiles
  • backlog and throughput charts for both runs
  • the classic-compute and Preview results, when you have them
  • a link to this thread

Please post back what Support finds. Others running small-file ingestion on serverless are going to run into this, and your thread is the best writeup of the symptom Iโ€™ve seen.

References:

Regards,
Louis

saurabh18cs
Honored Contributor III

Hi @rjmcbride what if you fiddle around wit max worksers for your serverless here 

saurabh18cs_0-1794562178613.png

 

@saurabh18cs , That is classic compute. Serverless does not allow the configuration of max workers.