cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
danatsafe
by • New Contributor
  • 5391 Views
  • 3 replies
  • 0 kudos

Amazon returns a 403 error code when trying to access an S3 Bucket

Hey! So far I have followed along with the Configure S3 access with instance profiles article to grant my cluster access to an S3 bucket. I have also made sure to disable IAM role passthrough on the cluster. Upon querying the bucket through a noteboo...

Data Engineering
Reply
  • 5391 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
winojoe
New Contributor III
  • 0 kudos

I had the same issue and I found a solutionFor me, the permission problems only exist when the Cluster's (compute's) Access mode is "Shared No Isolation".  When the Access Mode is either "Shared" or "Single User" then the IAM configuration seems to a...

  • 0 kudos
2 More Replies
gdoron
by • New Contributor
  • 1534 Views
  • 2 replies
  • 0 kudos

using pyspark can I write to an s3 path I don't have GetObject permission to?

After spark finishes writing the dataframe to S3, it seems like it checks the validity of the files it wrote with: `getFileStatus` that is `HeadObject` behind the scenes.What if I'm only granted write and list objects permissions but not GetObject? I...

Data Engineering
Reply
  • 1534 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Lakshay
Databricks Employee
  • 0 kudos

It is not possible in my opinion.

  • 0 kudos
1 More Replies
Jyo
by • New Contributor II
  • 2028 Views
  • 2 replies
  • 0 kudos

Rollback error - during deploying the workspace through quickstart

These are the steps I followed:1) Under quickstart-> added workspace name-> selected N-virginia (us-east-1), and quick start2) next step: except the password, I haven't edited any of the below:stackname- (default) databricks-workspace-stackparameters...

Data Engineering
Reply
  • 2028 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
rams
Contributor
  • 0 kudos

Hi Jyo, I followed the same "quickstart" approach to configure the databricks with aws account and got the same errors. Please can you tell us how to solve the issue if you have already solved. Thank you.

  • 0 kudos
1 More Replies
Juniper_AIML
by • New Contributor
  • 2778 Views
  • 1 replies
  • 0 kudos

How to setup Instance profile for initializing Databricks Cluster using Docker?

I was trying to start of the Databricks cluster through a docker image. I followed the setup instruction. Excluding the additional setup to setup the IAM role and instance profile as I was facing issues.The image is stored on AWS ECR in a public repo...

Screenshot 2022-02-15 at 2.39.57 PM Screenshot 2022-02-15 at 2.49.03 PM
Data Engineering
Reply
  • 2778 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hello again, @Aman Gaurav​. Thanks for your second question. As before, we'll see what the community has to say first.

  • 0 kudos
brickster_2018
by • Databricks Employee
  • 1823 Views
  • 1 replies
  • 0 kudos

Resolved! Getting file permission issues even though I have the right IAM role attached

I am reading data from S3 from a Databricks cluster and the read operation seldom fails with 403 permission errors. Restarting the cluster fixes my issue.

Data Engineering
Reply
  • 1823 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
brickster_2018
Databricks Employee
  • 0 kudos

The main reason for this behavior is : AWS keys are used in addition to the IAM role. Using global init scripts to set the AWS keys can cause this behavior.The IAM role has the required permission to access the S3 data, but AWS keys are set in the Sp...

  • 0 kudos
Labels