cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

For the Autoloader, cloudFiles.includeExistingFiles option, is ordering respected?

User16869510359
Esteemed Contributor

If Yes, how is order ensured?  For example, let's say there are a number of CDC change files that are uploaded to a directory over time. If a table were to be created using the cloudFiles source, in what order would those files be processed?

1 ACCEPTED SOLUTION

Accepted Solutions

User16869510359
Esteemed Contributor

Auto-loader does not ensure ordering of the files. You may feel the ordering is respected, but it could be just a coincidence.

View solution in original post

3 REPLIES 3

User16869510359
Esteemed Contributor

Auto-loader does not ensure ordering of the files. You may feel the ordering is respected, but it could be just a coincidence.

Hanish_Goel
New Contributor II

Hi, Is there any new development in terms of ensuring ordering of the files in autoloader?

ZAIvR
New Contributor II

You could use the new (currently in preview) Delta Live Table CDC feature to do this assuming there's a field in the files that indicates the proper sequence. See this article