topic Re: Does Databricks support AWS S3 Express One Zone? in Data Engineering

Does Databricks support AWS S3 Express One Zone?

cristianc — Fri, 15 Nov 2024 08:28:23 GMT

Greetings,

I'm writing this message since I learned that AWS has a storage class that is faster than S3 Standard called "S3 Express One Zone". (https://aws.amazon.com/s3/storage-classes/express-one-zone/)

AWS offers support for this storage class with AWS EMR and similar services (https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-express-one-zone.html)

The connector that in theory should offer the support for this is the hadoop s3a connector (https://issues.apache.org/jira/browse/HADOOP-18996) from version 3.3.7+, however I seen that in the latest databricks runtime 16, the hadoop-client-runtime is on 3.3.6, therefore I assume this is not supported.

In addition I have tried to map the s3 express bucket, also called a "directory bucket", in unity catalog, but unity catalog failed to recognize it as a valid bucket.

Has anyone else experimented with this?

Regards,

Cristian

Re: Does Databricks support AWS S3 Express One Zone?

Walter_C — Fri, 15 Nov 2024 15:39:02 GMT

Right now there is no support for S3 Express One Zone but this is already in our radar through idea DB-I-8058, this is currently tagged as Considered for the future, there is no ETA but our teams are working to have this supported in the near future.

Re: Does Databricks support AWS S3 Express One Zone?

cristianc — Mon, 12 May 2025 09:32:58 GMT

Are there any updates on DB-I-8058? Perhaps can you advice how can I track the progress on this?

Re: Does Databricks support AWS S3 Express One Zone?

marioquark — Mon, 16 Mar 2026 13:13:05 GMT

I also watch at this feature