I'm trying to do something similar either in a DLT pipeline or a standard streaming query autoloader job.  This thread implies there is a way to pass multiple source directories via options to the readstream query while maintaining state all through one checkpoint.   this would greatly simplify my process if true but I have been unable to get it to work.  I've also considered your appraoch which is to parameterize my file_path using subfolders as sources.