Hi, @SP_6721 and @Pawan_1 ,
Thanks for your concerns. I was able to find the problem. So the problem was the cluster was assigned to groups and for the permissions were given to individual users. When the permissions were given to the group, we were able to access the data. Not sure why it was happening as even in the groups, individual users were there so according to me this shouldn't have happened. But anyways the issue is solved. If any of you guys know why did it not work even when individual user had the permissions but still it shows permission error please enlighten me as even in the databricks documentations, was not able to find anything.