Databricks Autoloader processing old files

Prashanth24
New Contributor III

‎04-30-2025 12:38 AM

I have implemented Databricks Autoloader and found that every time i executes the code, it is still reading all old existing files + new files. As per the concept of Autoloader, it should read and process only new files. Below is the code. Please help me to understand what might have went wrong

df = (

spark.readStream.format("cloudFiles")

.option("cloudFiles.format","csv")

.option("cloudFiles.schemaLocation",<ADLS_PATH>)

.option("cloudEvolutionMode","rescue")

.option("header",True)

.load("abfss://container2@storageaccount1.dfs.core.windows.net/autoloader/input1/*/")

.writeStream

.option("checkpoint_location","abfss://container2@storageaccount1.dfs.core.windows.net/autoloader/checkpoint1")

.option("mergeSchema",True)

.trigger(availableNow=True)

.toTable("uniform_catalog1.autoloader2.table1")

)

0 Kudos

RameshChejarla
New Contributor III

‎04-30-2025 02:36 AM

Hi Prashanth, Auto loader for me its reading only new files, can you pls go through the below script.

df = (
spark.readStream
.format("cloudFiles")
.option("cloudFiles.format", "csv")
.option("cloudFiles.schemaLocation", "path")
.option("recursiveFileLookup", "true")
.option("header", "true")
.schema(schema)
.load("path")
)

df.writeStream \
.format("delta") \
.option("checkpointLocation", "path") \
.option("mergeSchema", "true") \
.trigger(availableNow=True) \
.toTable("path")

0 Kudos

Prashanth24
New Contributor III
In response to RameshChejarla

‎04-30-2025 03:36 AM

Thanks for the reply Ramesh. Have you tried executing display(df) after below code and see the results. Both of our code is almost similar and when i tried to see the results of df, contents of old processed files and new files were getting displayed

df = (
spark.readStream
.format("cloudFiles")
.option("cloudFiles.format", "csv")
.option("cloudFiles.schemaLocation", "path")
.option("recursiveFileLookup", "true")
.option("header", "true")
.schema(schema)
.load("path")
)

0 Kudos