For the Autoloader, cloudFiles.includeExistingFiles option, is ordering respected?

brickster_2018
Databricks Employee
Databricks Employee

‎06-25-2021 06:14 AM

If Yes, how is order ensured?  For example, let's say there are a number of CDC change files that are uploaded to a directory over time. If a table were to be created using the cloudFiles source, in what order would those files be processed?

Labels:
0 Kudos

brickster_2018
Databricks Employee
Databricks Employee

‎06-25-2021 06:17 AM

Auto-loader does not ensure ordering of the files. You may feel the ordering is respected, but it could be just a coincidence.

View solution in original post

0 Kudos

Hanish_Goel
New Contributor II

‎07-21-2022 08:32 AM

Hi, Is there any new development in terms of ensuring ordering of the files in autoloader?

0 Kudos

ZAIvR
New Contributor II
In response to Hanish_Goel

‎12-19-2022 06:02 AM

You could use the new (currently in preview) Delta Live Table CDC feature to do this assuming there's a field in the files that indicates the proper sequence. See this article

0 Kudos