-werners-
Esteemed Contributor III

If you do not have control over the content of the files I suggest the following:
Each day you get new files/data (I suppose these are not incremental).  These files contain new, updated and deleted data, and are overwritten.
Because of this, autoloader will not work (and copy into).
So the easiest way is to read the files and merge them into a delta lake table.  The merge statement will apply an insert/update or delete depending on your merge condition.
Because you do not use any fancy file detection mechanism, this will work.
For autoloader to work, you need unique filenames f.e. (aka no overwrites).
DLT might be able to do this too, but I don't use it so can't say a lot about that.

PS. we also never use SQL for engineering loads, only for analytical purposes.  Dataframes are a bliss according to us 🙂

View solution in original post