I'm using this code to read data from an ADLS Gen2 location. There are txt files present in sub-folders in the container.

    df_stream = spark.readStream \
        .format("cloudFiles") \
        .option("cloudFiles.format", "text") \
        .option('cloudFiles.includeExistingFiles', "false") \
        .option('cloudFiles.modifiedAfter', '2025-09-09 00:00:00.000000 UTC+0') \
        .format("text") \
        .load(LANDED_PATH) 

 All the old files in the location I need to skip as they have been processed previously. I followed the documentation and used includeExistingFiles  and modifiedAfter  option but these aren't working. The old files are still getting processed. Why are these options not working?