NandiniN
Databricks Employee
Databricks Employee

Meanwhile, on digging further 

Autoloader, in general, is highly recommended for ingestion where the files are immutable. While there are configurations (i.e. cloudFiles.allowOverwrites = True) that allow for updated files to be re-ingested in the source, AL only guarantees only once semantics when allowOverwrites is not enabled.

Can you please help me with two details 

- which DBR are you are on (please try with latest)

- Are you using Photon?