Saritha_S
Databricks Employee
Databricks Employee

Hi @FAHADURREHMAN 

Please find below my findings 

Since you're using Auto Loader (cloudFiles)  this behavior is expected.

By default, when you provide a path like:

s3://bucket/folder/

Spark recursively reads all subfolders.
pathGlobFilter="*.csv" only filters file names — it does NOT prevent recursive directory traversal.

To overcome the issue, please use the below 

Use recursiveFileLookup = false

.option("recursiveFileLookup", "false")

 or 

Use Explicit Wildcard Instead of Folder

s3://bucket/folder/*.csv

View solution in original post