madhav_dhruve
Databricks Partner

HI, I was able to solve the issue using single user cluster. It has necessary priviliges to access local file system. If you want to use it with shared cluster, you need an init script giving permissions to a new folder created by you inside temp directory.

Example - 

#!/bin/bash
sudo iptables -A INPUT -s 169.254.169.254 -j ACCEPT
sudo iptables -A OUTPUT -d 169.254.169.254 -j ACCEPT
mkdir -p /tmp/myfiles
chmod -R 0777 /tmp/myfiles

View solution in original post