Does Databricks support AWS S3 Express One Zone?

[cristianc]

Greetings,

I'm writing this message since I learned that AWS has a storage class that is faster than S3 Standard called "S3 Express One Zone". (https://aws.amazon.com/s3/storage-classes/express-one-zone/)

AWS offers support for this storage class with AWS EMR and similar services (https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-express-one-zone.html)

The connector that in theory should offer the support for this is the hadoop s3a connector (https://issues.apache.org/jira/browse/HADOOP-18996) from version 3.3.7+, however I seen that in the latest databricks runtime 16, the hadoop-client-runtime is on 3.3.6, therefore I assume this is not supported.

In addition I have tried to map the s3 express bucket, also called a "directory bucket", in unity catalog, but unity catalog failed to recognize it as a valid bucket.

Has anyone else experimented with this?

Regards,

Cristian

[Walter_C · Accepted solution]

Right now there is no support for S3 Express One Zone but this is already in our radar through idea DB-I-8058, this is currently tagged as Considered for the future, there is no ETA but our teams are working to have this supported in the near future.