Thanks comb for the suggestion and the sample code. Looking into the foreachbatch processing, this seems like if could work. My one concern with it is the information here about foreachbatch serializing the processing and removing parallelization, but this would probably be acceptable given the expected data sizes of the batched exports (a few dozen files, under 100000 records a batch).