Autoloader start and end date for ingestion
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-21-2023 06:04 AM
I have been searching for a way to set up backfilling using autoloader with an option to set a "start_date" or "end_date". I am working on ingesting a massive file system but I don't want to ingest everything from the beginning. I have a start date that I want to perform the first big ingestion to populate the most recent data into my database and then over time slowly backfill the older data. Is this functionality currently in the autoloader settings, and if not, any suggestions on how to approach this issue?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-06-2024 12:42 PM
If the files have already been loaded by autoloader (like same name and path), this can be tricky.
I recommend starting a separate autoloader stream and specifying filters on it to match your start and end dates. If you'd instead like to rely on the modification timestamps of the files, you can use the modifiedBefore and modifiedAfter options.
