- 9028 Views
- 6 replies
- 6 kudos
Resolved! What does autoloader's cloudfiles.backfillInterval do?
I'm using autoloader directory listing mode (without incremental file listing) and sometimes, new files are not picked up and found in the cloud_files-listing.I have found that using the 'cloudfiles.backfillInterval'-option can resolve the detection ...
- 9028 Views
- 6 replies
- 6 kudos
- 6 kudos
If we set the backfill to 1 week, will it run only 1ce a week or rather it will look for old files not processed in every trigger ?For eg :- if we set it to 1 day and the job runs every hour, then will it look for files in past 24 hours on a sliding ...
- 6 kudos
- 7888 Views
- 5 replies
- 1 kudos
Resolved! Autoloader directory listing not listing all files
Hi communityI have an Autoloader pipeline running with following configuration. Unfortunately, it does not detect all files. (see below query definition). The folder that needs to be read has 38.246 files that all have the same schema and structure.:...
- 7888 Views
- 5 replies
- 1 kudos
- 1 kudos
Hi @Fabrice Deseyn​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answer...
- 1 kudos