Hi @somanath Sankaran​ ,

I will recommend to use trigger.AvailableNow instead of trigger.once. Here is the link to the docs https://docs.databricks.com/release-notes/runtime/10.1.html#triggeravailablenow-for-auto-loader

Goin back to your original question, you can use incremental listing.  partitions can be considered lexically ordered if data is processed once a day, file paths containing timestamps can be considered lexically ordered.

Docs here https://docs.databricks.com/spark/latest/structured-streaming/auto-loader-gen2.html#incremental-list...