Hi @ChristianRRL ,

It looks like spark.readStream with Auto Loader creates a continuous streaming job by default, which means it keeps running while waiting for new files.

To avoid this, you can control the behaviour using trigger(availableNow=True), which processes all data available at the start, but may break the work into multiple micro-batches.