Good series. The query-scoped vs driver-scoped framing from Part 1 is something I haven’t seen written down clearly before, even though everyone who’s run streaming in prod has hit it.
One thing that kept nagging me reading all three parts: Serverless Jobs never comes up. That’s the obvious answer to “cost is why we haven’t switched.” Per-task isolation, no cluster lifecycle to manage, no cold start tax. What was the reason it was off the table?
Also the ConcurrentAppendException mention at the end of Part 3 is the thing I most want to read about. That’s not a retry problem, that’s Delta isolation levels and isBlindAppend semantics inside foreachBatch. Different beast entirely.