Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-06-2024 12:56 PM - edited 12-06-2024 12:57 PM
To answer the original question, autoloader does not use locks when reading files. You are however limited by the underlying storage system, ADLS in this example.
Going by what has been mentioned (long batch times, but spark jobs finish really fast) it sounds like you are limited by listing the directory. For high volume setups where source directories are not cleaned up of old files, we recommend using file notification mode - this works well as it avoids listing historical directories to find new files