Fair point - if it's genuinely append-only with no corrections, you're right, you're already touching the minimum amount of data each run. So, the problem really is what you originally said: how do you process a genuinely huge incoming batch efficiently, not how do you avoid touching old data.

One thing I'd still throw in for that specific case though - small files. If you're writing 2 billion new rows a month in an append-only pattern, you can end up with a ton of tiny files piling up in each partition even if everything else is tuned well. And that comes back to bite you later, because whoever queries this next month, including your own pipeline, ends up slower just from having to open and read through way more files than it needs to.

A couple of things that help here: coalescing or repartitioning right before the write so you're not writing one tiny file per task and then running OPTIMIZE on a schedule afterward to merge whatever small files still build up over time. If you're on a newer Delta version, auto-compaction and optimized writes do a lot of this automatically without you having to schedule anything extra. Either way, I'd add file size management as one more thing on your list, specifically for pipelines that are append-heavy like this one.