Permission denied during write

Daan · ‎01-27-2025

Hey everyone,

I have a pipeline that fetches data from s3 and stores them under the Databricks .tmp/ folder.
The pipeline is always able to write around 200 000 files before I get a Permission Denied error. This happens in the following code block:

os.makedirs(f".tmp/{filename_base[:-4]}", exist_ok=True).

There are no duplicates in filename_base[:-4].

Any idea why that is the case?

Thanks!