Alberto_Umana
Databricks Employee
Databricks Employee

Hi @subhas_hati,

Thanks for your question:

  1. Batch Workload: The availableNow trigger is used for batch processing. When you set the trigger to availableNow, it processes all available data as a single batch and then stops. This is useful for scenarios where you want to process all the data available at a specific point in time and then terminate the job.
  2. Streaming Workload: The processingTime trigger is used for streaming workloads. When you set the trigger to a specific time interval (e.g., processingTime='10 seconds'), it processes data in micro-batches at the specified interval. This allows for continuous processing of incoming data in near-real-time.

In your code, the appendInvoices method distinguishes between batch and streaming workloads based on the trigger parameter:

  • If trigger is set to "batch", it uses the availableNow trigger.
  • Otherwise, it uses the processingTime trigger with the specified interval.