COPY INTO maintaining row order

[wilsmith]

I have a CSV file in S3 and loading the rows in the order they appear in the file is necessary for parsing it out later. When using COPY INTO will it maintain that order so the bronze layer is in exactly the same order as the source file?

[Isi]

Hey @wilsmith 

COPY INTO does not guarantee the order of rows because it processes files in parallel using Spark’s distributed architecture. This means that the ingestion engine reads different parts of the file simultaneously, potentially splitting and reordering data for performance optimization. As a result, the original row order from the CSV file may not be preserved in the target table.

If maintaining the exact row order is important, the recommended approach is to add an explicit row_number column to the file before ingestion. This way, you can later sort the data using that column to reconstruct the original order reliably.

Hope this helps, 🙂

Isi