I have a DLT pipeline that reads data in S3 into an append-only bronze layer using Autoloader. The data sink needs to be changed to a new s3 bucket in a new account, and data in the existing s3 bucket migrated to the new one.

Will Autoloader still be able to tell that it has already processed those files after they have been replicated to the new S3 bucket?

I'm thinking of setting Autoloader's `cloudfiles.includeExistingFiles=False` and the table property `{"pipelines.reset.allowed": "false"}` to avoid re-processing all that old Bronze layer data if not. Sample Python code below, for reference:

@dlt.table(
    name=f"append_only_bronze_layer",
    table_properties={"quality": "bronze"},
)
def raw_bronze_layer() -> DataFrame:
    return (
        spark.readStream.format("cloudFiles")
        .options(
            cloudFiles.format="avro",
            cloudFiles.inferColumnTypes=True,
        )
        .load(/this/path/will/change)
    )