cgrant
Databricks Employee
Databricks Employee

To answer the original question, autoloader does not use locks when reading files. You are however limited by the underlying storage system, ADLS in this example.

Going by what has been mentioned (long batch times, but spark jobs finish really fast) it sounds like you are limited by listing the directory. For high volume setups where source directories are not cleaned up of old files, we recommend using file notification mode - this works well as it avoids listing historical directories to find new files