Prabakar
Databricks Employee
Databricks Employee

I suspect the problem could be caused by the cluster mode. What type of cluster or what access mode do you use for your cluster?