Hi @sanjay, Looking to effectively manage your autoloader's continuous trigger? Follow these steps for seamless execution:
Pausing the Trigger at Specific Times: If you need to halt the continuous trigger during certain hours, consider switching to a triggered pipeline. However, if you prefer to stick with the continuous trigger, programmatically stop it through the workspace client or the Job Rest API1. Just be sure to handle this approach carefully to avoid any data loss.
To keep track of the number of records still waiting in the pipeline, your best bet is to keep an eye on the associated Structured Streaming job for your autoloader. While there isn't a direct method to retrieve the exact count, you can gain insight by monitoring the job's progress and checkpoint files. It's worth noting that if the job gets halted during a micro-batch, any changes to the trigger interval won't take effect until that batch is complete. This can therefore serve as a useful gauge for determining the presence of pending records. Just bear in mind that Databricks Auto Loader operates using Structured Streaming, so familiarizing yourself with how triggers function will ultimately allow for better control over costs and optimal data ingestion.
