Mounting of S3 bucket via Terraform is frequently timing out

165036 — Tue, 26 Jul 2022 06:09:31 GMT

Summary of the problem

When mounting an S3 bucket via Terraform the creation process is frequently timing out (running beyond 10 minutes). When I check the Log4j logs in the GP cluster I see the following error message repeated:

```

22/07/26 05:54:43 ERROR DatabricksS3LoggingUtils$:V3: S3 request failed with com.amazonaws.SdkClientException: Unable to execute HTTP request: Remote host terminated the handshake; Request ID: null, Extended Request ID: null, Cloud Provider: AWS, Instance ID: i-0af1e2435799123e0

com.amazonaws.SdkClientException: Unable to execute HTTP request: Remote host terminated the handshake

...

Caused by: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake

...

Caused by: java.io.EOFException: SSL peer shut down incorrectly

at sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:481)

at sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:470)

at sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:160)

at sun.security.ssl.SSLTransport.decode(SSLTransport.java:110)

at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1418)

... 89 more

```

It seems like the `databricks_mount` is a flaky and works sometimes and not others.

What I've tried

I've ensured that the instance profile role attached to the general purpose cluster nodes have the recommended policy with s3:ListBucket, s3:PutObjectAcl, s3:PutObject, s3:GetObject and s3:DeleteObject permissions.
I've also completely removed the bucket policy to rule out that it's the bucket policy that's blocking the access

Please let me know if you need any further details.

Re: Mounting of S3 bucket via Terraform is frequently timing out

165036 — Mon, 01 Aug 2022 03:44:28 GMT

Solved. See here: https://github.com/databricks/terraform-provider-databricks/issues/1500

topic Mounting of S3 bucket via Terraform is frequently timing out in Data Engineering

Mounting of S3 bucket via Terraform is frequently timing out

Re: Mounting of S3 bucket via Terraform is frequently timing out