Hi, @anasimiao  

I did find a workaround, but ended up not using the modifiedBefore option in Auto Loader. Using modifiedBefore would have been the cleanest and easiest solution, but it doesn't seem to be intended for the purpose of reading files after they have reached a certain age (if they have already been ignored by an earlier run of Auto Loader).

In my case, every new CSV file gets written to a timestamped folder. I start by listing out all of these folders using 

folders = dbutils.fs.ls("path-to-folder-containing-timestamped-folders")

Then I convert each one to datetime format and keep only the ones older than 5 minutes. (Actually, in my case I know that all folders except for the most recent one contain CSV files that are done being written to, so I simply sort the folders by timestamp and remove the most recent one).

I then construct a string that looks like this:

timestamped_folders = "{2024-10-30T11.51.47Z,2024-10-30T11.56.48Z}" 
storage_path = f"path-to-folder-containing-timestamped-folders/{timestamped_folders}" 
... 
df = spark.readStream.format("cloudFiles") 
     ... 
     .load(storage_path)
     ...

The storage_path string is constructed using the curly brackets glob pattern so that Auto Loader will only consider files in either of the two timestamped folders "2024-10-30T11.51.47Z" or "2024-10-30T11.56.48Z", and not in any other folder. Read more about this here.

As a result, Auto Loader is restricted from considering files in the most recent timestamped folder (which I want it to ignore). The next time my Databricks job runs and more recent timestamped folders are written to my cloud storage, Auto Loader will be able to consider CSV files in the timestamped folder which used to be the most recent one.

This is a bit of a hacky/manual approach, but I hope it's clear enough. As a note, I initially wanted to use a glob pattern to specify which folder to ignore (the most recent one), instead of which folders to consider (all but the most recent one), because it would be a lot simpler. However, if a new folder appears in my cloud storage between the initial run of 

folders = dbutils.fs.ls("path-to-folder-containing-timestamped-folders")

and the run of Auto Loader, then I would be telling Auto Loader to ignore the second most recent folder, because a new folder was just written to cloud storage. So, as a safeguard against this, I decided to explicitly tell Auto Loader which folders it could consider, instead of telling Auto Loader which folder it couldn't consider.