Thank for the info. We have auto loader running daily just fine all the files are in delta. My question is more on the aws side as there no clear best practice for sourcing data from s3 then moving the files to glacier without moving them to another bucket. The s3 with delta and other files in the workspace are completely separate from this.

I think the two directory approach would work but don’t have an easy way to test this approach.

Removing the files from the source that auto loader reads from is a must as the job takes proportionally longer to run as the amount of files grow. I also build the schema from reading over the files in the s3 source bucket. With the 20million files present it’s requires a lot of compute resources than if there were a couple million. Each json file contains 260 fields at a very deep level. I have changes being made to the files soon and need the bucket simplified for new data being written for auto loader to read and for aws to move after a couple days into glacier objects as backup in case we need them again.