Real time pipelines

KKo — Fri, 31 Jul 2026 20:24:06 GMT

How do you design high-volume real-time pipelines that need to stay reliable under load spikes?

Re: Real time pipelines

ThomazNeto — Fri, 31 Jul 2026 20:57:55 GMT

Big question, but in my experience it comes down to five decisions. Opinionated take:

1. Put a buffer in front, always. Kafka/Kinesis/Event Hubs between producers and Spark. The broker absorbs the spike; your pipeline drains at its own pace. A pipeline that ingests directly from producers has no shock absorber — that's the #1 design mistake I see.

2. Cap what each batch is allowed to swallow. maxOffsetsPerTrigger (Kafka) or maxFilesPerTrigger/maxBytesPerTrigger (Auto Loader). Without a cap, a spike turns into one giant batch that blows memory or runs forever; with it, batches stay uniform and latency degrades gracefully instead of the job falling over. Backlog grows, then drains — that's the behavior you want.

3. Be honest about your latency SLO, because it picks the architecture. If seconds-to-minutes is fine, classic micro-batch absorbs spikes beautifully (backlog + recovery) and you can autoscale. If you genuinely need milliseconds, real-time mode went GA this year — but read the requirements carefully: classic compute only, autoscaling OFF, Photon off, no spot instances. No autoscaling means you provision for peak, not average. That's the price of ms latency, and it changes your whole capacity math under spikes.

4. Bound your state. Stateful ops (dedup, joins, sessionization) are what actually die under load — state grows unbounded without watermarks. Use RocksDB as the state store, set realistic watermarks, and if you have complex state logic, transformWithState gives you explicit control including TTL.

5. Monitor the two numbers that predict the incident: batch duration vs trigger interval, and source backlog/lag. When batch duration creeps toward the interval, you're saturated — alert there, not when it's already on fire. And make the sink idempotent, because under spikes you WILL get retries.

If you share your source, latency target and rough events/sec, happy to get more specific — the right answer for 10k events/s with a 5-minute SLO and for 500k/s with a 200ms SLO are two different pipelines.

topic Real time pipelines in Data Engineering

Real time pipelines

Re: Real time pipelines