Auto loader

[BhaveshPatel]

Suppose I have 1000's of historical .csv files stored from Jan, 2022 in a folder of my azure blob storage container. I want to use auto loader to read files beginning only on 1st, Oct, 2023 and ignoring all the files before this date to build a pipeline to read multiple files that are ingested daily.
Is this possible using Auto loader, if yes, how?
0 responses . Be the first to respond

[daniel_sahal]

@BhaveshPatel 

Three things that you can do:

- Move the files to the separate folder,
- Use a filter on metadata fields to filter out the unnecessary files,
- Use a pathGlobFilter to filter only on the files you need