Auto loader
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-28-2024 06:08 PM
Suppose I have 1000's of historical .csv files stored from Jan, 2022 in a folder of my azure blob storage container. I want to use auto loader to read files beginning only on 1st, Oct, 2023 and ignoring all the files before this date to build a pipeline to read multiple files that are ingested daily.
Is this possible using Auto loader, if yes, how?
0 responses . Be the first to respond
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-04-2024 12:59 AM
Three things that you can do:
- Move the files to the separate folder,
- Use a filter on metadata fields to filter out the unnecessary files,
- Use a pathGlobFilter to filter only on the files you need
