szymon_dybczak
Esteemed Contributor III

Conceptually they're the same. So they will load all available data. But the implementation differs.
In the case of trigger.Once, Spark Structured Streaming will try to load all available data in a single micro-batch. As you can imagine, if there’s a very large amount of data, this can cause serious issues. That’s why this option is deprecated.
trigger.AvailableNow will also load all available data, but by using a series of micro-batches. In the end, the effect will be the same, but with AvailableNow you won’t risk crashing the cluster when trying to load a massive amount of data 🙂

View solution in original post