cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

For the Autoloader, cloudFiles.includeExistingFiles option, is ordering respected?

User16869510359
Esteemed Contributor

If Yes, how is order ensured?  For example, let's say there are a number of CDC change files that are uploaded to a directory over time. If a table were to be created using the cloudFiles source, in what order would those files be processed?

1 ACCEPTED SOLUTION

Accepted Solutions

User16869510359
Esteemed Contributor

Auto-loader does not ensure ordering of the files. You may feel the ordering is respected, but it could be just a coincidence.

View solution in original post

3 REPLIES 3

User16869510359
Esteemed Contributor

Auto-loader does not ensure ordering of the files. You may feel the ordering is respected, but it could be just a coincidence.

Hanish_Goel
New Contributor II

Hi, Is there any new development in terms of ensuring ordering of the files in autoloader?

ZAIvR
New Contributor II

You could use the new (currently in preview) Delta Live Table CDC feature to do this assuming there's a field in the files that indicates the proper sequence. See this article

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.