topic Re: How to create an external location that accesses a public s3 bucket in Data Engineering

How to create an external location that accesses a public s3 bucket

deano2025 — Thu, 08 May 2025 08:45:16 GMT

Hi,

I'm trying to create an external location that accesses a public s3 bucket (for open data). However, I'm not having any success. I'm confused to what to specify as the storage credential (IAM role) since its a public bucket that is out of my control. By the way, I can easily select the data directly using pyspark e.g. by calling spark.read.json, but thought it made sense to use an external location first. Any ideas on the steps to take? Or is using an external location a waste of time in this case?

Thanks!

Re: How to create an external location that accesses a public s3 bucket

Isi — Thu, 08 May 2025 12:12:47 GMT

Hey @deano2025

Using an external location for a public S3 bucket can be unnecessarily

External locations are designed to:

Govern access to private cloud storage (S3, ADLS, GCS)
Map Unity Catalog permissions to cloud-level security via storage credentials
Work with managed tables, volumes, delta sharing, etc.

They rely on a Storage Credential, which is usually an IAM role or access key that grants access to a private bucket

But if you still want to create an External Location you can create a dummy Storage Credential using a placeholder ARN like:

arn:aws:iam::123141241214124:role/role_test

Then, use that credential when defining your External Location.

However, since the bucket is public, you’ll get the same result as simply reading it directly with spark.read...

Hope this helps to clarify, 🙂

Isi

Re: How to create an external location that accesses a public s3 bucket

deano2025 — Fri, 09 May 2025 10:23:02 GMT

Thanks @Isi Now that you've explained external locations, I think it does indeed make sense that they are probably unnecessary in this case. Thanks for clarifying!