Hi,
Is there any way to exclude a file from the dlt pipeline (autoload) run temporarily? What I mean is that I want to be able to exclude a specific file until I decided to include it in the load? I can't control the files or the location where they are stored.
Assume we have the following:
folder/file1
folder/file2
folder/file3
so, I want to run the dlt pipeline only for file 1 and 2. in the future I want to include file 3. I can exclude the file using some filters or parameters like ModifiedBefore...etc and right now the dlt pipeline reads all files and loads file 1 and 2 successfully. However, when I remove my exclusion criteria from the script. the dlt pipeline doesn't include the excluded file, I think the pipeline does register it somewhere or keeps track of all files and doesn't consider it as a new file.
Please let me know if I need to clarify anything further.
I appreciate any help or advice.