FileNotFoundError: [Errno 2] No such file or directory: '../00_configuration/prd/main_configuration.

[AdrianLobacz]

Maybe someone has encountered this problem before?

I’m running parallel loading for 10 objects using pool.map. Nine of them complete successfully, but one fails when trying to read a configuration file. The problem occurs occasionally and doesn’t follow any regular pattern.  All of them read the same configuration file. After rerunning the job, the issue did not occur again and all objects completed successfully. 

I’ve seen a similar issue before when releasing new changes to the dev environment. However, this time it’s happening in production, there was no deployment during that period, and the other 9 objects ran without any issues.

The error I’m getting is:

FileNotFoundError: [Errno 2] No such file or directory: '../00_configuration/prd/main_configuration.yaml'

What could be causing this problem?

[balajij8]

@AdrianLobacz 

You can read the configuration once and pass the object into your function instead of reading the same file multiple times. It eliminates the IO overhead and avoids hitting the FUSE layer. When the code triggers parallel processes, they hit the FUSE layer simultaneously. FUSE fails to refresh the metadata for one specific thread occasionally resulting in an error even though the file is there. You can pass the absolute path instead of relative path.