topic Re: For the Autoloader, cloudFiles.includeExistingFiles option, is ordering respected? in Data Engineering

For the Autoloader, cloudFiles.includeExistingFiles option, is ordering respected?

brickster_2018 — Fri, 25 Jun 2021 13:14:28 GMT

If Yes, how is order ensured? For example, let's say there are a number of CDC change files that are uploaded to a directory over time. If a table were to be created using the cloudFiles source, in what order would those files be processed?

Re: For the Autoloader, cloudFiles.includeExistingFiles option, is ordering respected?

brickster_2018 — Fri, 25 Jun 2021 13:17:09 GMT

Auto-loader does not ensure ordering of the files. You may feel the ordering is respected, but it could be just a coincidence.

Re: For the Autoloader, cloudFiles.includeExistingFiles option, is ordering respected?

Hanish_Goel — Thu, 21 Jul 2022 15:32:04 GMT

Hi, Is there any new development in terms of ensuring ordering of the files in autoloader?

Re: For the Autoloader, cloudFiles.includeExistingFiles option, is ordering respected?

ZAIvR — Mon, 19 Dec 2022 14:02:06 GMT

You could use the new (currently in preview) Delta Live Table CDC feature to do this assuming there's a field in the files that indicates the proper sequence. See this article