LuizFbm2002
New Contributor II

I believe this happens because of the checkpoint that delta streams store. When you process data directly from S3 via autoloader, it's reading the files from the bucket. In the silver/gold layer, you start using the stream/delta engine; since there's a checkpoint where data processing stopped, the stream discards what's already been processed and only processes new data. I believe that if you're only going to do this once, you should reset the checkpoint. Or model the pipeline with a CDC structure, since the solution from the colleague above of processing everything doesn't fit your scenario.