Hello @victorNilsson 

Thank you for letting me know how to replicate the issue, I was able to get the same error this time. 

I've given the problem another go and think I have been able to fix it by specifying the output path as "/tmp/test.csv". By writing to "/tmp" I'm hoping that this will still work as you initially intended: the file should not persist across clusters or sessions.

I tested this approach on both your latest and original code and it seemed to run successfully for both.

Based on my research, I believe this is why you were getting your issue:

By not defining a path, the notebook was trying to write to the default DBFS. DBFS is a distributed filesystem, and writing to this means the file is stored in cloud-backed storage instead of a local disk. I believe my initial solution failed because the function os.fsync requires lower-level disk features (that term may mean more to you than me but I understand it to mean that direct interaction with hardware operations is required). This also explains why mmap was causing the errors, as this too requires the same features.

The reason it was working across two cell is still likely related to the sequencing. Once a cell has finished the file is fully written to DBFS allowing it to then be re-opened in the following cell. 

I hope this helps with your issue, please let me know how you get on.

Regards - Pilsner