Databricks Autoloader processing old files

[Prashanth24]

I have implemented Databricks Autoloader and found that every time i executes the code, it is still reading all old existing files + new files. As per the concept of Autoloader, it should read and process only new files. Below is the code. Please help me to understand what might have went wrong

df = (

spark.readStream.format("cloudFiles")

.option("cloudFiles.format","csv")

.option("cloudFiles.schemaLocation",<ADLS_PATH>)

.option("cloudEvolutionMode","rescue")

.option("header",True)

.load("abfss://container2@storageaccount1.dfs.core.windows.net/autoloader/input1/*/")

.writeStream

.option("checkpoint_location","abfss://container2@storageaccount1.dfs.core.windows.net/autoloader/checkpoint1")

.option("mergeSchema",True)

.trigger(availableNow=True)

.toTable("uniform_catalog1.autoloader2.table1")

)

[RameshChejarla]

Hi Prashanth, Auto loader for me its reading only new files, can you pls go through the below script.

df = (
spark.readStream
.format("cloudFiles")
.option("cloudFiles.format", "csv")
.option("cloudFiles.schemaLocation", "path")
.option("recursiveFileLookup", "true")
.option("header", "true")
.schema(schema)
.load("path")
)

df.writeStream \
.format("delta") \
.option("checkpointLocation", "path") \
.option("mergeSchema", "true") \
.trigger(availableNow=True) \
.toTable("path")

[Prashanth24]

Thanks for the reply Ramesh. Have you tried executing display(df) after below code and see the results. Both of our code is almost similar and when i tried to see the results of df, contents of old processed files and new files were getting displayed

df = (
spark.readStream
.format("cloudFiles")
.option("cloudFiles.format", "csv")
.option("cloudFiles.schemaLocation", "path")
.option("recursiveFileLookup", "true")
.option("header", "true")
.schema(schema)
.load("path")
)