forEachBatch is an option, but then the merge will take a long time (merge per file).

Also (I forgot to mention that): a single file can also contain multiple versions of a single record.

Not using autoloader seems the way to go at the moment, but it would be nice if it is possible after all without a lot of overhead.