Hi,

I see you are using `Trigger.AvailableNow`. Is this intended to be a continuous stream or an incremental batch trigger at an interval with Databricks Workflows?

From the docs (https://docs.databricks.com/en/structured-streaming/triggers.html#configure-structured-streaming-tri...

> Databricks recommends you use Trigger.AvailableNow for all incremental batch processing workloads.
> The available now trigger option consumes all available records as an incremental batch with the ability to configure batch size with options such as maxBytesPerTrigger (sizing options vary by data source).

If intending to run in continuous mode, can you please try with a different trigger interval?

Thanks.