Its preferable to use spark streaming (with Delta) for batch workloads rather then regular batch. With the trigger.once trigger whenever the streaming job is started it will process whatever is available in the source (kafka/kinesis/File System) and keep track of the progress in the streaming checkpoint location. So after it succeeds when you run the job again it will leverage checkpoint to know where to start from.