Unable to access S3 objects from Databricks using IAM access keys in both AWS and Azure Databricks

Obulreddy
Databricks Partner

‎07-06-2023 05:36 AM

Hi Team,

We are trying to connect to Amazon S3 bucket from both Databricks running on AWS and Azure using IAM access keys directly through Scala code in Notebook and we are facing com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden; with status code as 403. But by using the same credentials we were able to access it from AWS CLI.

Sample code that we ran in Notebook

 

import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder
  .appName("S3 Access")
  .getOrCreate()
// Set AWS access key and secret key
spark.conf.set("spark.hadoop.fs.s3a.access.key", "***********")
spark.conf.set("spark.hadoop.fs.s3a.secret.key", "**********************")
// Set the S3 endpoint URL if needed
spark.conf.set("spark.hadoop.fs.s3a.endpoint", "s3.us-east-2.amazonaws.com")
// Read a file from the S3 bucket into a DataFrame
val df = spark.read.parquet("s3://<parquet_file_path>")
df.show()
 
 
Thanks,
Obul.
0 Kudos

KaKa
Contributor

‎07-16-2023 11:21 PM

need to create instance profile. read more details following url : https://docs.databricks.com/aws/iam/instance-profile-tutorial.html

hope it work 🙂

0 Kudos

Anonymous
Not applicable

‎07-20-2023 12:35 AM

Hi @Obulreddy 

We haven't heard from you since the last response from @KaKa ​, and I was checking back to see if her suggestions helped you.

Or else, If you have any solution, please share it with the community, as it can be helpful to others. 

Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.

KaKa
Contributor
In response to Anonymous

‎07-20-2023 12:43 AM

Hi @Anonymous 

from my site clearly, no more concerns

tks very much

0 Kudos