Databricks Community

Dusko · ‎02-03-2022

Hi, I’m trying to read file from S3 root bucket. I can ls all the files but I can’t read it because of access denied. When I mount the same S3 root bucket under some other mountPoint, I can touch and read all the files. I also see that this new mountPoint have an empty encryptionType while others (including root) have sse-s3.

Maybe this could be the issue (we are trying access data through instance profiles)? My goal is to make root mountPoint "/" accessible because of all our hardcoded paths in dependencies and modules. We already checked all IAM roles, instance_profiles, bucket policies but without any luck.

Additional info:

Files I want to read are uploaded manually through AWS console
When I import .zip file as .whl library and %sh unzip that file I can read this file without any problem

Is there something we are missing? Thank you for your help!

Anonymous · ‎02-03-2022

Hi @Dušan Vystrcil. My name is Piper, and I'm one of the moderators here in the community. Thanks for your question and it's great to meet you. 🙂

We will give the community a chance to respond and then we will come back around if we need to.

Atanu · ‎02-12-2022

As databricks has the ownership of rootbucket I think manually upload from console will not work.For your second point - ( import .zip file as .whl library and %sh unzip ) I hope you are doing this through notebook so it will not giving you any issue.

Anonymous · ‎02-16-2022

@Dušan Vystrcil - Checking in. Does Atanu's answer help you?

Dusko · ‎02-17-2022

Hi @Atanu Sarkar , @Piper Wilson ,

thanks for the replies. Well I don't understand the fact about ownership. I believe that rootbucket is still under my ownership (I created it and I could upload/delete any files through browser without any problem). I can even list those files from notebook but can't read them.

I discovered that if I upload those files through databricks cli, I could read them. So maybe you're right about the "ownership" but I can't find any docs regarding this, so I could understand it. Do we have anything?