Thanks for the reply Ramesh. Have you tried executing display(df) after below code and see the results. Both of our code is almost similar and when i tried to see the results of df, contents of old processed files and new files were getting displayed

df = (
spark.readStream
.format("cloudFiles")
.option("cloudFiles.format", "csv")
.option("cloudFiles.schemaLocation", "path")
.option("recursiveFileLookup", "true")
.option("header", "true")
.schema(schema)
.load("path")
)