cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

FabriceDeseyn
by Contributor
  • 3419 Views
  • 5 replies
  • 6 kudos

Resolved! What does autoloader's cloudfiles.backfillInterval do?

I'm using autoloader directory listing mode (without incremental file listing) and sometimes, new files are not picked up and found in the cloud_files-listing.I have found that using the 'cloudfiles.backfillInterval'-option can resolve the detection ...

image
  • 3419 Views
  • 5 replies
  • 6 kudos
Latest Reply
Kiranrathod
New Contributor III
  • 6 kudos

Hi @Lakshay Goel​ ,where can I set the backFillInterval property in the code? Do you have any sample codes for this use case?

  • 6 kudos
4 More Replies
FabriceDeseyn
by Contributor
  • 3043 Views
  • 5 replies
  • 1 kudos

Resolved! Autoloader directory listing not listing all files

Hi communityI have an Autoloader pipeline running with following configuration. Unfortunately, it does not detect all files. (see below query definition). The folder that needs to be read has 38.246 files that all have the same schema and structure.:...

image.png image.png image.png image.png
  • 3043 Views
  • 5 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Fabrice Deseyn​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answer...

  • 1 kudos
4 More Replies
Labels