topic Unable to Access S3 from Serverless but Works on Cluster in Data Engineering

Unable to Access S3 from Serverless but Works on Cluster

HarryRichard08 — Mon, 10 Mar 2025 17:34:59 GMT

Hi everyone,

I am trying to access data from S3 using an access key and secret. When I run the code through Databricks clusters, it works fine. However, when I try to do the same from a serverless cluster , I am unable to access the data.

I have already checked:

The credentials (access key and secret) are correct.
databricks scope

Could someone guide me on what additional configurations or steps I need to follow to make this work from a server?

Any help is greatly appreciated!

Thanks!

Re: Unable to Access S3 from Serverless but Works on Cluster

KaranamS — Mon, 10 Mar 2025 19:22:13 GMT

Hi @HarryRichard08 , Can you please provide the sample code you are using? And the error you get with serverless?

Re: Unable to Access S3 from Serverless but Works on Cluster

HarryRichard08 — Tue, 11 Mar 2025 11:20:08 GMT

Hi Karanam

Please find the syntax below

spark.conf.set("fs.s3a.access.key", "xxxxxxxxxxxxxxxxxxxx")

spark.conf.set("fs.s3a.secret.key", "xxxxxxxxxxxxxxxxxxxxxx")

Re: Unable to Access S3 from Serverless but Works on Cluster

KaranamS — Tue, 11 Mar 2025 21:38:16 GMT

Hi @HarryRichard08, Databricks recommends using instance profiles (IAM roles) to connect to AWS S3 as they provide a secure and scalable method without embedding credentials in a notebook. Have you tried this approach?

https://docs.databricks.com/aws/en/connect/storage/tutorial-s3-instance-profile