topic Re: Databricks Autoloader processing old files in Data Engineering

Databricks Autoloader processing old files

Prashanth24 — Wed, 30 Apr 2025 07:38:46 GMT

I have implemented Databricks Autoloader and found that every time i executes the code, it is still reading all old existing files + new files. As per the concept of Autoloader, it should read and process only new files. Below is the code. Please help me to understand what might have went wrong

df = (

spark.readStream.format("cloudFiles")

.option("cloudFiles.format","csv")

.option("cloudFiles.schemaLocation",<ADLS_PATH>)

.option("cloudEvolutionMode","rescue")

.option("header",True)

.load("abfss://container2@storageaccount1.dfs.core.windows.net/autoloader/input1/*/")

.writeStream

.option("checkpoint_location","abfss://container2@storageaccount1.dfs.core.windows.net/autoloader/checkpoint1")

.option("mergeSchema",True)

.trigger(availableNow=True)

.toTable("uniform_catalog1.autoloader2.table1")

)

Re: Databricks Autoloader processing old files

RameshChejarla — Wed, 30 Apr 2025 09:36:54 GMT

Hi Prashanth, Auto loader for me its reading only new files, can you pls go through the below script.

df = (
spark.readStream
.format("cloudFiles")
.option("cloudFiles.format", "csv")
.option("cloudFiles.schemaLocation", "path")
.option("recursiveFileLookup", "true")
.option("header", "true")
.schema(schema)
.load("path")
)

df.writeStream \
.format("delta") \
.option("checkpointLocation", "path") \
.option("mergeSchema", "true") \
.trigger(availableNow=True) \
.toTable("path")

Re: Databricks Autoloader processing old files

Prashanth24 — Wed, 30 Apr 2025 10:36:20 GMT

Thanks for the reply Ramesh. Have you tried executing display(df) after below code and see the results. Both of our code is almost similar and when i tried to see the results of df, contents of old processed files and new files were getting displayed