Hi @sakuraDev ,

1. Using the availableNow trigger to process all available data immediately and then stop the query. As you noticed your data was processed once and now you need to trigger the process once again to process new files.

2. Changing the trigger to .trigger(processingTime="1 second") means that the streaming query will attempt to process any new files every second. If there are no new files due to a source outage, the query will not terminate; it will continue to check for new files at the specified interval.

Important consideration

The cluster would be running continuously. This means much bigger costs compared to running the process with availableNow trigger.

3. Don't worry about the outage. The whole idea of autoloader is the checkpointing mechanism. If you stop or detach the Auto Loader job and then restart it, the job will resume processing from where it left off.. The checkpointLocation option you've specified allows Auto Loader to keep track of which files have been processed. When the job is restarted, it will process any new files that arrived during the outage, ensuring no data is missed.