We have a set of streaming dlt pipelines following a medallion pattern where s3 bucket -> autoloader -> bronze delta tables -> silver delta tables -> gold delta tables. All delta tables are in a unity catalog under separate schemas. We need a solution to be able to delete a set of data records from gold, silver and bronze and reload, by renaming or copying the original files again to the s3 bucket. When we attempted this, the autoloader does pickup the renamed files and reload to the bronze delta table, but does not load further to silver and gold delta tables. Is this expected behavior ? Why does the silver delta table stream reader not able to pickup the newly reloaded data in the bronze delta table ? There is no error, but its seems to completely ignore that reloaded data, but picks up other newly arrived data. We want to have this ability to delete and reload data again using the same pipelines without writing completely separate pipelines with this work. Is there any solution for this ?
Thanks in advance
Ravi Pal