For the Autoloader, cloudFiles.includeExistingFiles option, is ordering respected?

[brickster_2018]

If Yes, how is order ensured?  For example, let's say there are a number of CDC change files that are uploaded to a directory over time. If a table were to be created using the cloudFiles source, in what order would those files be processed?

[brickster_2018 · Accepted solution]

Auto-loader does not ensure ordering of the files. You may feel the ordering is respected, but it could be just a coincidence.

[Hanish_Goel]

Hi, Is there any new development in terms of ensuring ordering of the files in autoloader?

[ZAIvR]

You could use the new (currently in preview) Delta Live Table CDC feature to do this assuming there's a field in the files that indicates the proper sequence. See this article