How Auto Loader works – file level or row level?

Akshay_Petkar — Tue, 29 Jul 2025 15:52:20 GMT

Does Auto Loader work on file level or row level? If it works on file level and does not process the same file again, then how can we make it pick only the new rows when data is appended to that file?

Re: How Auto Loader works – file level or row level?

szymon_dybczak — Tue, 29 Jul 2025 16:13:31 GMT

Hi @Akshay_Petkar ,

Autoloader works on file level. Now, by default autoloader is configured with following option:

cloudFiles.allowOverwrites = false

So, above option causes files to be processed exactly once.

But when you switch this option to true, then Auto Loader is guaranteed to process the latest version of the file. But keep in mind that autloader will reprocess entire file (even if there was partial update).
You can read detail description of this behaviour here:

Auto Loader FAQ - Azure Databricks | Microsoft Learn

topic How Auto Loader works – file level or row level? in Data Engineering

How Auto Loader works – file level or row level?

Re: How Auto Loader works – file level or row level?