One pipeline woke up every five minutes. The other stayed running.
Using the same e-commerce streaming workload, median end-to-end data freshness improved from 4.0 minutes to 33 seconds, while P95 latency dropped from 9.3 minutes to just 60 seconds. Those results were expected for a continuously running pipeline. What wasn't expected was what we found inside the Serverless Spark Declarative Pipeline itself: driver logs showed that eligible streaming queries could have two or more micro-batches in flight simultaneously.
That behavior is enabled by stream pipelining, a Serverless Spark Declarative Pipelines (SDP) optimization that overlaps work across successive micro-batches instead of waiting for one batch to fully complete before beginning the next. For workloads where processing time exceeds the configured trigger interval, this can significantly improve resource utilization and reduce end-to-end latency while preserving the familiar Spark Structured Streaming programming model.
What you'll learn
- When to choose Triggered or Continuous mode based on workload characteristics and data freshness requirements.
- How stream pipelining works internally and how it differs from the traditional sequential execution model of Spark Structured Streaming.
- What we observed in production-style benchmarks and driver logs, including evidence of overlapping micro-batches and the resulting impact on latency and throughput.
The Setup
To make the comparison concrete, We built an e-commerce order-processing benchmark that represents workloads that can have both periodic analytics requirements and near-real-time operational requirements such as fraud detection and live revenue monitoring.
- Producer: ~5,000 orders/min (~15,000 line items/min after explode)
- Kafka: Amazon MSK cluster with TLS, connected through PrivateLink in a AWS us-west-2 region. Both Databricks and Kafka cluster running in the same region.
- Compute: Serverless SDP for both pipelines using defaults settings.
- Pipeline architecture at high level: Bronze → Silver → Gold
- Transforms: 25+ identical transformations in the Silver layer
- Test conducted during the 07/13/2026 EST timezone.
Each order contains a nested items array with 1–5 items (about three on average), so explode() amplifies the data volume by roughly 3×. The Silver layer applies SHA-256 hashes, regex extraction, multi-factor fraud scoring, state-based tax calculation, shipping-zone classification, price normalization, and other transformations.
The primary execution difference is that one pipeline runs in Triggered mode on a five-minute schedule, while the other remains active in Continuous mode and both pipelines read from the same kafka topic concurrently. The Continuous pipeline uses a 1-second trigger interval for Bronze and Silver and a 30-second interval for Gold.
Triggered vs. Continuous: The First Architectural Decision
Before looking at stream pipelining, it is important to separate two questions. The first is a pipeline lifecycle decision: should the pipeline wake up periodically, process available data, and stop—or should it remain active and continuously process new data?
Triggered Mode: Process, Stop, and Start Again
In Triggered mode, each scheduled invocation runs a pipeline lifecycle: compute is made available, the pipeline initializes, sources are connected, available data is processed, progress is committed, and the run terminates. The pipeline then waits until the next scheduled invocation and data that is continuously arrived after pipeline starts will wait for the next update.
In our benchmark, a full Triggered lifecycle took 153–194 seconds, averaging about 177 seconds per invocation. With a five-minute schedule, this created a pattern of active processing followed by idle time before the next scheduled run. In this case each cycle includes: provisioning serverless compute, JVM initialization, pipeline graph setup, Kafka consumer group creation, data processing, checkpoint commit, and teardown.
Continuous Mode: Start Once and Keep Processing
In Continuous mode, the pipeline starts and remains active. New data is processed through successive micro-batches according to the configured trigger intervals, and these trigger intervals can be configured at pipeline level or at each flow/table level, without repeating the full pipeline startup and teardown lifecycle for every scheduled processing window.
For workloads with continuously arriving data, continuous mode provides a more consistent execution model while avoiding repeated lifecycle overhead. In our benchmark, the Silver layer processed approximately 1,400 line items per batch across more than 25 transformations. The roughly 766 ms lightweight-processing time shown in the example above is intended only to illustrate how continuous mode works; it does not represent the actual Silver-layer pipeline used in the benchmark below.After the one-time startup, micro-batches fire continuously with sub-second processing times. No cold start, no teardown, no wasted compute.
When Should You Use Each Mode?
The choice should be driven by workload characteristics rather than by assuming that Continuous mode is always better.
|
Consideration |
Triggered Mode |
Continuous Mode |
|
Data arrival |
Periodic, bursty, hourly, or daily |
Continuous or frequent event arrival |
|
Freshness requirement |
Minutes to hours can be acceptable |
Sub-minute to few-minute freshness |
|
Idle periods |
Long idle periods between arrivals |
Steady or frequent incoming data |
|
Pipeline lifecycle |
Starts and stops for each invocation |
Remains active |
|
Typical use cases |
Periodic ETL, hourly fulfillment, daily reporting |
Fraud detection, live dashboards, inventory tracking, CDC |
|
Primary trade-off |
Avoid unnecessary active execution during idle periods which eventually saves the execution cost. |
Consistent low-latency processing and steady-state execution |
Triggered mode therefore remains a valid—and often preferable—choice when data arrives infrequently or when latency is not critical. Continuous mode becomes more compelling when data arrives continuously and downstream consumers expect consistently fresh results.
Benchmark Result: The Lifecycle Choice Affects Data Freshness
We measured end-to-end data freshness as processed_at - order_timestamp: the time from when an order was created until it appeared in the enriched Silver table.
For this workload and five-minute Triggered schedule, Continuous mode delivered substantially fresher data. That result is useful, but it is not the entire story. Once Continuous mode is selected, Serverless SDP introduces another optimization worth understanding: Stream Pipelining.
Going Deeper: What Is Different Inside Continuous Mode?
This is where the second comparison begins. Triggered versus Continuous describes the pipeline lifecycle. Stream pipelining describes how eligible micro-batches can be executed within a continuously running Serverless SDP pipeline.
To understand why this matters, consider a simple question: what happens when a streaming query has a 1-second trigger interval, but each micro-batch takes more than one second to complete?
Classic Spark Structured Streaming: Sequential Micro-Batch Execution
In the traditional Spark Structured Streaming micro-batch execution pattern, a streaming query processes micro-batches sequentially. Spark still parallelizes tasks and stages within each micro-batch, but successive micro-batches of the same query do not independently execute as fully overlapping batches.
An important distinction: Spark task parallelism within a micro-batch is not the same thing as micro-batch pipelining across batches. The comparison here is specifically about whether work associated with Batch N+1 can overlap with work from Batch N.
If a micro-batch takes longer than the configured trigger interval, the next batch must wait for the current batch to finish before it can start
The above figure illustrates exactly how standard Spark Structured Streaming executes micro-batches strictly one at a time. Each micro-batch must complete all phases—Plan, Execute, Write, and Commit—before the next micro-batch can begin, even if the configured trigger interval is shorter than the batch processing time. As a result, the effective trigger interval increases from 1,000 ms to 1,450 ms, leaving the compute idle between batches and reducing overall resource utilization and throughput.
Certain resources are underutilized during phases (writing to Delta, committing checkpoints) that do not saturate the cluster, pipelining can use otherwise available capacity for a subsequent batch.
Serverless SDP: Stream-Pipelined Micro-Batch Execution
Databricks docs describe it simply:
"Instead of running microbatches sequentially like standard Spark Structured Streaming, serverless Lakeflow Spark Declarative Pipelines runs microbatches concurrently, improving compute resource utilization. Stream pipelining is enabled by default in serverless pipelines."
But what does "concurrently" actually mean? Let's visualize it.
For eligible streaming queries, Serverless Lakeflow Spark Declarative Pipelines can use stream pipelining. Instead of requiring strictly serial end-to-end completion of successive micro-batches, the engine can overlap eligible work across batches.
Conceptually, while an earlier batch is completing later work such as writing and committing, a subsequent batch can already be progressing. The pipeline remains micro-batch based; stream pipelining adds another dimension of execution concurrency.
The key insight: while batch 1 is writing data to Delta and committing the checkpoint, batch 2 is already reading from source and executing its transformations. The CPU cores that would be idle during I/O wait are now processing the next batch.
How Deep Can Pipelining Go?
Our benchmark showed that overlap was not limited to two batches. In the Silver query, we observed cases where up to three micro-batches were in flight simultaneously.
The Silver layer was configured with a 1-second trigger interval, while its average batch duration was 2,242 ms. This created sustained pressure where processing duration exceeded the configured cadence—exactly the scenario where pipelined execution becomes particularly relevant.
Going Under the Hood: What Do the Driver Logs Tell Us?
So far, we’ve seen conceptually how stream pipelining allows micro-batches to overlap. Now, let’s go one level deeper and look at what actually happens under the hood.
We analyzed approximately 230 MB of driver logs from a 30-minute Continuous-mode benchmark run to understand how Serverless SDP schedules and overlaps micro-batches during execution. SDP engine logs specific strings that encode its internal decisions and per-batch timings. The method was to grep those out and count/bucket them.
Pipelining Is Enabled at the Streaming Query Level:
The engine logs an explicit status message for each streaming query at startup:
INFO MicroBatchExecution: [queryId = c831c] Pipelined execution is enabled
for query c831c9dc-073f-4d03-876f-85c7e23114d2.
Reason:
isServerless = true
pipeliningEnabledInServerless = true
isStateful = false <-- stateless = eligible
deltaSinkWithCompleteMode = false <-- append mode = eligible
sinkSupport = true
sourcesSupport = true
isPipeliningForceDisabled = false
isStatefulPipeliningForceDisabled = false
sameDeltaSourceSink = false
All four streaming queries in the benchmark logged pipelining as enabled, including the Gold query. Eligibility is evaluated per query.
What Micro-Batch Overlap Looks Like in Practice:
The micro-batch pipelining progress metrics give us an even deeper view into what is happening during execution. In the Silver query, 99.7% of the observed batches showed overlap. Most had two micro-batches in flight, while 6.1% showed three micro-batches in flight simultaneously.
|
Query |
No overlap |
2 in flight |
3 in flight |
|
Silver |
1 (0.3%) |
308 (93.6%) |
20 (6.1%) |
|
Bronze |
5 (1.6%) |
315 (98.4%) |
0 |
|
Gold |
60 (100%) |
0 |
0 |
|
Metrics |
312 (99.4%) |
2 (0.6%) |
0 |
Here's the raw log entry showing 3 concurrent batches in silver:
[queryId = c831c] [batchId = 96] Streaming query made progress:
"timestamp" : "2026-07-13T02:30:59.729Z", <-- batch 96 STARTS
"batchDuration" : 2270, <-- runs until 02:31:01.999Z
"numInputRows" : 106
[queryId = c831c] [batchId = 97] Streaming query made progress:
"timestamp" : "2026-07-13T02:31:00.729Z", <-- batch 97 STARTS (1.0s later)
"batchDuration" : 2457, <-- runs until 02:31:03.186Z
"numInputRows" : 394
Batch 97 begins 1.0s after batch 96, but batch 96 keeps running for another 1.27s. Their execution windows overlap by 1,270 ms: two micro-batches processing concurrently against a 1s trigger.
Batch Duration vs. Trigger Interval
|
Query |
Trigger |
Avg Batch Duration |
Exceeds Trigger |
Batches |
|
Silver |
1s |
2,242 ms (2.2×) |
100% |
329 |
|
Bronze |
1s |
1,421 ms (1.4×) |
99.7% |
320 |
|
Gold |
30s |
7,885 ms (0.26×) |
0% |
60 |
|
Metrics |
1s |
2,387 ms (2.4×) |
96.5% |
314 |
The Silver query is the clearest example: average processing duration was more than twice the trigger interval, and nearly every observed batch showed overlap. The Gold query, by contrast, completed well within its 30-second interval and showed no observed batch overlap in this run.
Cross-Table Parallelism: The Other Dimension
- Across tables: different tables in the pipeline can execute as independent streaming queries.
- Within a query: eligible successive micro-batches can overlap through stream pipelining.
These dimensions are in addition to Spark's normal parallel execution of tasks within an individual micro-batch.
Which Queries Can Benefit from Pipelining?
Not every streaming query is guaranteed to use stream pipelining. The engine evaluates eligibility independently for each query.
An interesting finding in our benchmark was that the Gold query in this environment logged pipelining enabled despite state, then note it showed no actual overlap because its duration was below the trigger. It used a watermark with approx_count_distinct and append-mode output. This suggests that eligibility should be understood at the query level rather than reduced to a simple rule that all stateful queries are excluded.
The Complete Picture
The complete comparison therefore has two levels. First, choose Triggered or Continuous mode based on workload arrival patterns, latency requirements, idle time, and operational goals. Second, when Continuous mode is the right choice, Serverless SDP can provide an additional execution advantage through stream pipelining for eligible queries.
Practical Recommendations
Use Triggered Mode When:
- Data arrives in infrequent or predictable bursts, such as hourly or daily loads.
- There are long idle periods between data arrivals.
- Processing latency is not critical and several minutes—or longer—is acceptable.
- The workload is periodic ETL or reporting rather than continuous operational processing.
- Avoiding active pipeline execution during long idle windows is an important consideration.
Use Continuous Mode When:
- Data arrives continuously from sources such as Kafka, CDC feeds, or event streams.
- Sub-minute to few-minute freshness matters.
- The workload supports operational use cases such as fraud detection, live dashboards, or inventory tracking.
- A steady-state pipeline is preferable to repeated startup and teardown cycles.
- After Continuous is justified by the freshness SLO, eligible queries may gain additional utilization/throughput benefits from stream pipelining.
When Tuning Continuous Pipelines:
- Choose trigger intervals based primarily on freshness requirements and workload characteristics—not simply to force overlap.
Warning : The reason we choose 1 second interval for Bronze and Silver tables is to showcase this overlapping, but do not shorten the trigger merely to force overlap. The success criterion is meeting the freshness SLO without sustained backlog or instability, not maximizing the number of concurrent batches.
- Monitor whether batch processing duration consistently exceeds the configured trigger interval.
- Use pipeline metrics and logs to understand whether pipelining is active for individual queries.
- Remember that task-level Spark parallelism, cross-table parallelism, and micro-batch pipelining are separate dimensions of concurrency.
- Use pipelines.trigger.interval per table to match processing complexity
Conclusion
Triggered and Continuous modes solve different problems. Triggered mode makes sense for periodic or bursty workloads where latency requirements are relaxed and the pipeline can remain inactive between processing windows. Continuous mode is better suited to continuously arriving data and workloads that require consistently fresh results.
But for Continuous workloads, Serverless Lakeflow Spark Declarative Pipelines introduce another important consideration: stream pipelining. Rather than following only the traditional sequential micro-batch execution pattern, eligible queries can overlap work across successive micro-batches, which eventually improves overall performance and throughput by using those idle CPU cycles.
In our benchmark, the Silver query averaged 2,242 ms per batch against a 1-second trigger interval. We observed overlap in 99.7% of Silver batch observations, including cases with three micro-batches in flight simultaneously improving the throughput by ~1.7X. The driver logs and micro-batch progress metrics provided direct evidence of that behavior.
The takeaway is not that Continuous mode is always better than Triggered mode. The right mode depends on the workload. But when Continuous processing is the right architectural choice, Serverless SDP's stream pipelining can provide an additional execution advantage—helping eligible streaming queries overlap micro-batch work and use available compute more effectively when processing pressure exceeds the configured trigger cadence.
2 Comments
