All great points! A few more follow-up questions.

I can see the benefits to using AutoLoader for large volume real time or near real time streaming data (readStream/writeStream under the hood after all). But I'm wondering, is this "overkill" for data ingestion that is more batch-oriented?

For example, while we do have some real or near real time use cases, a majority of our use cases are relating to either event or timeseries (10-min/15-min/hourly) data that we are typically ok to pull daily. While "streaming" the data is not strictly necessary, I see that I can effectively run it as a batch via the trigger option on writeStream:

.trigger(availableNow=True)

Also, for some additional context, we are currently running with all-purpose compute clusters in dev/prod. Does AutoLoader have suggested/preferred computes to run with in order to save (or avoid) overspending on compute resources?