Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-24-2026 07:05 AM
Please find below my findings
Since you're using Auto Loader (cloudFiles) this behavior is expected.
By default, when you provide a path like:
s3://bucket/folder/
Spark recursively reads all subfolders.pathGlobFilter="*.csv" only filters file names — it does NOT prevent recursive directory traversal.
To overcome the issue, please use the below
Use recursiveFileLookup = false
.option("recursiveFileLookup", "false")
or
Use Explicit Wildcard Instead of Folder
s3://bucket/folder/*.csv