Once a week, I get very long run times with AutoLoader. The spark job says it is done, but garbage collection keeps rising on the driver. I assume this is because of the backfill interval that I am using with FileNotification Type. I have this set to do so every week.

Is my suspicion right? If so how could I handle this and is the Backfill Interval necessary?