Reading data from S3 in Azure Databricks

[ossinova]

Is it possible to create an external volume in Azure Databricks that points to an external S3 bucket so that I can read files for processing? Or is it only limited to ADLSv2?

[PiotrU]

I don't think so - but there are some other ways to "mount" it as external storage (some time ago I have used sc._jsc.hadoopConfiguration().set - not sure if It does still work ) or maybe things like s3fs ? 

[szymon_dybczak]

Hi @ossinova ,

I think currently it's not possible to create external volume to S3 bucket in UC. But still you can access S3 data using following techniques:

1. Access S3 buckets with URIs and AWS keys
2. Access S3 with open-source Hadoop options
3. You can mount S3 bucket (but this method is depracted)

Connect to Amazon S3 - Azure Databricks | Microsoft Learn

[Ashley1]

Yep, I'm keen to see this functionality as well.

I think it is reasonable to expect external locations can be on diverse storage types (at least the big players). I can nicely control access to azure storage in UC but not S3.