Streaming Doesn't Mean Your Compute Needs to Run Forever

kartheek_rao — Thu, 20 Aug 2026 16:07:57 GMT

When I first started working with streaming in Databricks, I had a very simple picture in my head:

Streaming = keep the cluster running and process data continuously.

New data arrives → Spark sees it → process it → repeat forever.

Technically, that can be streaming.

But while building a pipeline around GDELT data flowing through Azure Data Lake into Databricks, I started realizing something much more important:

The architecture of a streaming system should be driven by how quickly the business needs the data — not by the fact that the source keeps producing data.

That distinction completely changed the way I started looking at Structured Streaming.

Imagine my source receives a new file every few minutes.

Does the business really need the result within 2 seconds?

Maybe.

Fraud detection probably does.

Real-time personalization might.

Monitoring an industrial system might.

But what if I am processing news articles for NLP analysis?

If predictions become available 5, 10, or even 15 minutes later, does anything actually break?

Probably not.

And suddenly keeping compute alive continuously starts looking less like a technical requirement and more like an expensive architectural assumption.

This is where AvailableNow became interesting to me.

Instead of thinking:-> "My stream must always be running."

I can think -> "Process everything that has arrived since the previous run, maintain the state of the pipeline, and then stop."

Databricks Structured Streaming supports exactly this pattern with Trigger.AvailableNow().

It processes the available unprocessed data—potentially across multiple micro-batches—and then terminates. The checkpoint allows the next execution to continue from the appropriate position rather than blindly starting everything again.

That sounds like a small configuration choice.

But architecturally, it is much bigger.

It separates two concepts that I had initially mixed together:

Streaming data
and
always-on compute.

They are not necessarily the same thing.

Then checkpoints started making much more sense.

Initially, I looked at a checkpoint directory as just another configuration that Spark required.

Something we add because the documentation says so.

But a checkpoint is really the memory of the streaming application.

It helps the stream remember its progress and state.

Without that memory, restarting a pipeline safely becomes much harder.

With it, the system can essentially say:

"I've already dealt with this part of the stream. Let's continue from where we were."

Databricks explicitly describes the checkpoint as providing the unique identity for a stream while tracking processed records and associated state.

Once I understood that, checkpoints stopped feeling like infrastructure clutter.

They became part of the reliability architecture.

Micro-batches gave me another realization.

At first, the word "streaming" made me imagine one record travelling through the architecture at a time.

But Spark Structured Streaming often operates differently.

Records arriving during a period are grouped and processed incrementally as micro-batches.

And that gives us another architectural control point.

If incoming data suddenly increases, we don't necessarily want one enormous batch consuming everything the source can provide.

Databricks provides controls such as maxFilesPerTrigger and maxBytesPerTrigger for controlling how much Auto Loader processes per micro-batch. This can help keep resource consumption more predictable.

Again, this isn't just Spark configuration.

It is capacity planning.

And this is probably my biggest takeaway.

When learning platforms like Databricks, it is easy to spend too much time asking:

"Which API should I use?"

But the more useful questions are:

How fresh does my data actually need to be?

What happens when my pipeline stops?

How does it know what has already been processed?

What happens if 10 files suddenly become 10,000 files?

Am I paying for latency that nobody actually needs?

These questions lead naturally to concepts such as Auto Loader, checkpoints, trigger strategies, admission controls, observability, and pipeline orchestration.

Auto Loader itself is designed to incrementally discover and process new files arriving in cloud object storage, and Databricks now recommends using it with Lakeflow pipelines for many production incremental-ingestion workloads.

So today, when I hear:

"We need streaming."

My next question is no longer:

"Which streaming API should we use?"

It is:

"What latency does the business actually need?"

Because sometimes the best streaming architecture isn't the one processing every second.

It is the one that knows when to process, what has already been processed, how much to process at once — and when to shut itself down.

That, for me, was the point where Structured Streaming started feeling less like a Spark feature and more like an architectural design problem.

Curious how others approach this:
For workloads where 5–15 minute latency is acceptable, do you prefer triggered incremental processing, or do you still have reasons to keep the pipeline continuously running?

topic Streaming Doesn't Mean Your Compute Needs to Run Forever in Data Engineering

Streaming Doesn't Mean Your Compute Needs to Run Forever

This is where AvailableNow became interesting to me.

Then checkpoints started making much more sense.

Micro-batches gave me another realization.

And this is probably my biggest takeaway.