Real time pipelines

[KKo]

How do you design high-volume real-time pipelines that need to stay reliable under load spikes?

[ThomazNeto]

Big question, but in my experience it comes down to five decisions. Opinionated take:

1. Put a buffer in front, always. Kafka/Kinesis/Event Hubs between producers and Spark. The broker absorbs the spike; your pipeline drains at its own pace. A pipeline that ingests directly from producers has no shock absorber — that's the #1 design mistake I see.

2. Cap what each batch is allowed to swallow. maxOffsetsPerTrigger (Kafka) or maxFilesPerTrigger/maxBytesPerTrigger (Auto Loader). Without a cap, a spike turns into one giant batch that blows memory or runs forever; with it, batches stay uniform and latency degrades gracefully instead of the job falling over. Backlog grows, then drains — that's the behavior you want.

3. Be honest about your latency SLO, because it picks the architecture. If seconds-to-minutes is fine, classic micro-batch absorbs spikes beautifully (backlog + recovery) and you can autoscale. If you genuinely need milliseconds, real-time mode went GA this year — but read the requirements carefully: classic compute only, autoscaling OFF, Photon off, no spot instances. No autoscaling means you provision for peak, not average. That's the price of ms latency, and it changes your whole capacity math under spikes.

4. Bound your state. Stateful ops (dedup, joins, sessionization) are what actually die under load — state grows unbounded without watermarks. Use RocksDB as the state store, set realistic watermarks, and if you have complex state logic, transformWithState gives you explicit control including TTL.

5. Monitor the two numbers that predict the incident: batch duration vs trigger interval, and source backlog/lag. When batch duration creeps toward the interval, you're saturated — alert there, not when it's already on fire. And make the sink idempotent, because under spikes you WILL get retries.

If you share your source, latency target and rough events/sec, happy to get more specific — the right answer for 10k events/s with a 5-minute SLO and for 500k/s with a 200ms SLO are two different pipelines.

[emma_s]

Hi,

This is a broad question without much detail. The first thing I'd want to understand is what is the definition of realtime that is required. People say realtime but often they can mean a variety of things:

• What is the latency requirement and where is it coming from and too? is it a stream based feed where you need to process the data and then feedback to another stream? Or do you need to write to structured source such as Lakebase or delta format?
• When we talk about latency do you mean sub second, sub 15 seconds, under a minute or some other variety? This will set whether you can do micro batching or need a true streaming pipeline. Micro batching is usually cheaper and simpler, plus cope better with traffic spikes. If its sub second then you're going to be needing to read from Kafka or event hubs, processing with spark RTM and then writing back to another stream. If its less than a minute, spark declarative pipelines with streaming mode can help you handle this and will cope well with spikes if configured correctly.
• You also need to think about what level of transformation you need to do, this will greatly impact latency.

Some doc links to help with this:

https://docs.databricks.com/aws/en/structured-streaming/real-time/concepts

https://community.databricks.com/t5/technical-blog/a-practitioner-s-guide-to-real-time-mode-on-spark...

https://docs.databricks.com/aws/en/structured-streaming/examples

If you want to provide some more info, happy to help further.

Thanks,

Emma

[KKo]

I have both needs

a) sub seconds b) with in 10 minutes