cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Trigger.once mode recommendation

User16783853906
Contributor II

When is it recommended to use Trigger.once mode compared to fixed processing intervals with micro batches?

2 REPLIES 2

User16783853906
Contributor II

With trigger.once mode, only one micro batch will be executed to process the latest data and application stops on its own once the processing is complete. This is helpful in cases where you need to periodically process everything that is available since the last processed data.

Fore more information on this, refer to - https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#triggers

User16869510359
Esteemed Contributor

Also note, the configurations like maxFilesPerTrigger, maxBytesPerTrigger are ignored with Trigger.Once.

Streaming queries with significantly less throughput can switch to Trigger.Once to avoid the continuous execution of the job checking the availability of the data.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.