Hi,

Is there any way to exclude a file from the dlt pipeline (autoload) run temporarily? What I mean is that I want to be able to exclude a specific file until I decided to include it in the load? I can't control the files or the location where they are stored.

Assume we have the following:

folder/file1

folder/file2

folder/file3

so, I want to run the dlt pipeline only for file 1 and 2. in the future I want to include file 3. I can exclude the file using some filters or parameters like ModifiedBefore...etc and right now the dlt pipeline reads all files and loads file 1 and 2 successfully. However, when I remove my exclusion criteria from the script. the dlt pipeline doesn't include the excluded file, I think the pipeline does register it somewhere or keeps track of all files and doesn't consider it as a new file.

Please let me know if I need to clarify anything further.

I appreciate any help or advice.