topic Re: Unable to access S3 objects from Databricks using IAM access keys in both AWS and Azure Databric in Get Started Discussions

Unable to access S3 objects from Databricks using IAM access keys in both AWS and Azure Databricks

Obulreddy — Thu, 06 Jul 2023 12:36:08 GMT

Hi Team,

We are trying to connect to Amazon S3 bucket from both Databricks running on AWS and Azure using IAM access keys directly through Scala code in Notebook and we are facing com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden; with status code as 403. But by using the same credentials we were able to access it from AWS CLI.

Sample code that we ran in Notebook

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder

.appName("S3 Access")

.getOrCreate()

// Set AWS access key and secret key

spark.conf.set("spark.hadoop.fs.s3a.access.key", "***********")

spark.conf.set("spark.hadoop.fs.s3a.secret.key", "**********************")

// Set the S3 endpoint URL if needed

spark.conf.set("spark.hadoop.fs.s3a.endpoint", "s3.us-east-2.amazonaws.com")

// Read a file from the S3 bucket into a DataFrame

val df = spark.read.parquet("s3://<parquet_file_path>")

df.show()

Thanks,

Obul.

Re: Unable to access S3 objects from Databricks using IAM access keys in both AWS and Azure Databric

KaKa — Mon, 17 Jul 2023 06:21:16 GMT

need to create instance profile. read more details following url : https://docs.databricks.com/aws/iam/instance-profile-tutorial.html

hope it work 🙂

Re: Unable to access S3 objects from Databricks using IAM access keys in both AWS and Azure Databric

Anonymous — Thu, 20 Jul 2023 07:35:49 GMT

Hi @Obulreddy

We haven't heard from you since the last response from @KaKa , and I was checking back to see if her suggestions helped you.

Or else, If you have any solution, please share it with the community, as it can be helpful to others.

Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.

Re: Unable to access S3 objects from Databricks using IAM access keys in both AWS and Azure Databric

KaKa — Thu, 20 Jul 2023 07:43:39 GMT

Hi @Anonymous

from my site clearly, no more concerns

tks very much