Have you tried writing to the local file system (for example, in the path /databricks/...​)

​

Anyway, i recommend you to tyr writing with Spark (without dataframes). Pandas without additional libraries, doesn't distribute and with high volumes you will have memory problems, performance problems...