Autoloader directory listing not listing all files

FabriceDeseyn
Contributor

Hi community

I have an Autoloader pipeline running with following configuration. Unfortunately, it does not detect all files. (see below query definition).

image.pngimage.png 

The folder that needs to be read has 38.246 files that all have the same schema and structure.:

image.pngIf I look at the `cloud_files_state`, I see get 4999 files. Do I do something wrong? Is this an 'initial' count that will be increased if these files are ingested?

image.png