Hi @Anastasiia Polianska​,

I agree, it looks like a concurrency issue. Very possibly this concurrency problem will be caused by an erroneous ETAG in the HTTP call to the Azure Storage API (https://azure.microsoft.com/de-de/blog/managing-concurrency-in-microsoft-azure-storage-2/)

The concurrency behavior can be configured according to the hadoop-azure library documentation. It is the library used to access ADLS (abfss)

https://hadoop.apache.org/docs/stable/hadoop-azure/abfs.html#Server_Options

Surely these links will help you understand/solve your problem:

• https://hadoop.apache.org/docs/stable/hadoop-azure/abfs.html

• GitHub: https://github.com/apache/hadoop/tree/rel/release-3.3.4/hadoop-tools/hadoop-azure

Thanks.

Fernando Arribas.