topic Re: Reading data from S3 in Azure Databricks in Data Engineering

Reading data from S3 in Azure Databricks

ossinova — Wed, 04 Sep 2024 11:44:59 GMT

Is it possible to create an external volume in Azure Databricks that points to an external S3 bucket so that I can read files for processing? Or is it only limited to ADLSv2?

Re: Reading data from S3 in Azure Databricks

PiotrU — Wed, 04 Sep 2024 12:33:18 GMT

I don't think so - but there are some other ways to "mount" it as external storage (some time ago I have used sc._jsc.hadoopConfiguration().set - not sure if It does still work ) or maybe things like s3fs ?

Re: Reading data from S3 in Azure Databricks

szymon_dybczak — Wed, 04 Sep 2024 13:15:22 GMT

Hi @ossinova ,

I think currently it's not possible to create external volume to S3 bucket in UC. But still you can access S3 data using following techniques:

Access S3 buckets with URIs and AWS keys
Access S3 with open-source Hadoop options
You can mount S3 bucket (but this method is depracted)

Connect to Amazon S3 - Azure Databricks | Microsoft Learn

Re: Reading data from S3 in Azure Databricks

Ashley1 — Tue, 17 Sep 2024 00:50:59 GMT

Yep, I'm keen to see this functionality as well.

I think it is reasonable to expect external locations can be on diverse storage types (at least the big players). I can nicely control access to azure storage in UC but not S3.