Thanks for the thoughtful perspective. I completely agree that the first question should be whether we need to process the full historical dataset or only what has changed.

In the scenario I had in mind, however, the incoming dataset consists entirely of new transaction records (an append-only feed), so the pipeline is already processing only the newly arrived data rather than reprocessing historical records each run.

In that case, the challenge shifts from reducing the data volume to efficiently processing billions of new records. That’s where partitioning, minimizing shuffles, selecting the right join strategy, handling data skew, optimizing file sizes, and tuning Spark execution become the primary focus.

I also agree with your point about late-arriving corrections. Once updates or corrections to historical data are introduced, the design becomes more complex and requires CDC, idempotent MERGE operations, and selective reprocessing of affected partitions rather than simple append logic.

So I think both approaches are complementary the first step is understanding the data ingestion pattern, and once we know we’re dealing with an append-only stream of new transactions, Spark execution and pipeline optimization become the key design considerations. Thanks for adding that perspective.