We have a data feed with files whose filenames stays the same but the contents change over time (brand_a.csv, brand_b.csv, brand_c.csv ....).

Copy Into seems to ignore the files when they change.

If we set the Force flag to true and run it, we end up with duplicates.

If you Truncate the table and attempt to re-load (without setting force to true), databricks doesn't re-copy the records.

So, behind the scenes, Databricks is holding some kind of load history info and is smart enough not to re-load files it's already seen but not quite smart enough to know that a file has been updated and potential updates are available.

We're pulling _metadata column in which lists filename, size, mod date, etc. Not that Databricks knows that but it does have enough information at some point to know when an input file has changed.

This particular dataset is small enough where dropping the table and recreating isn't that big a deal but I'm interested in knowing for the general case. 

Do either Autoloader or Delta Live handle that sort of situation differently or do I need to load into a fresh staging table and run MERGE INTO to handle it?

FWIW, we're using UNITY and DELTA.