Cloudformation error when launching Databricks in AWS

[BradSheridan]

I've seen many posts here in the Community as potential solutions to this error, but none seem to be a solution for us. We are trying to launch the 14 day free trial of Databricks from the AWS Marketplace and are getting the error below. Moreover, our partner manager at Databricks still hasn't replied to us, and neither have we received an answer when we emailed partners@databricks, so our hope is that someone here has the answer 🙂 Thanks!!

[ERROR] 2022-08-24T17:54:17.899Z ea746964-04dc-4796-9f5f-472f9b96b32d Exception: An error occurred (AccessDenied) when calling the CopyObject operation: Access Denied

Traceback (most recent call last):

File "/var/task/index.py", line 42, in handler

copy_objects(source_bucket, dest_bucket, prefix, objects)

File "/var/task/index.py", line 17, in copy_objects

s3.copy_object(CopySource=copy_source, Bucket=dest_bucket,

File "/var/runtime/botocore/client.py", line 391, in _api_call

return self._make_api_call(operation_name, kwargs)

File "/var/runtime/botocore/client.py", line 719, in _make_api_call

raise error_class(parsed_response, operation_name)

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the CopyObject operation: Access Denied

[Debayan]

This is a typical error where IAM role policies has to be checked. Could you please recheck the role policies and confirm?

[BradSheridan]

looking now...however, what exactly am I looking for in the policy to identify as potentially incorrect?

[BradSheridan]

Hi again @Debayan Mukherjee​. We're still struggling to get the cloudformation template deployed and I'm not getting much assistance from our partner manager. any chance you have availability for a brief call?

[Prabakar]

CopyObject operation: Access Denied -- looks like the permission to storage bucket is denied. Is this happening for all users or only while using specific IAM roles?

Also if you are using a cross-account role please check if the role is present in the bucket policy.

[BradSheridan]

thanks for the response @Prabakar Ammeappin​ . We have manually created the S3 bucket and are waiting for someone at our client site that has IAM priviledges (we are contractors to them and don't have that permission) to create the cross-account role. My hope is to report back here in a few hours with the solution 🙂

[BradSheridan]

Morning @Prabakar Ammeappin​ same question for you that I just posted to debayan....We're still struggling to get the cloudformation template deployed and I'm not getting much assistance from our partner manager. any chance you have availability for a brief call?

[BradSheridan]

@Debayan Mukherjee​ @Prabakar Ammeappin​ got passed the CopyObject error message and now the CloudFormation template errors out near the end on 'createWorkspace'. Any thoughts?

[Prabakar]

Hi @Brad Sheridan​ our time zone are different and i am unable to respond ​to you immediately. It's good to know you resolved the copyobject issue. Please provide the steps that you used to resolve the issue so it would help others.

For the error at create workspace, what is the error you are facing? Please add the screenshot with the error.

[BradSheridan]

Hi @Prabakar Ammeappin​ and thanks for getting back to me. The createWorkspace error happens near the end of the Cloudformation template and here is the error:

Received response status [FAILED] from custom resource. Message returned: Conflict InputData: {'workspace_name': 'nsc-poc-workspace', 'aws_region': 'us-east-1', 'credentials_id': '1f6914f7-9c47-4bf3-a1ed-c87c8451bea0', 'storage_configuration_id': '8ecf6b3e-3e6d-4629-af96-8cb70311bad1', 'network_id': '259c58e2-78f8-4285-b798-69126d87133c', 'deployment_name': 'nsc-poc'} (RequestId: d52f686d-d71a-4fd7-b15f-c15ec16da57e)

The following resource(s) failed to create: [createWorkspace].

[BradSheridan]

regarding the copyObject error that we overcame, I'll ask one of my engineers on Monday how he got passed the error and report back here.

[Prabakar]

Hi @Brad Sheridan​  Could you please try creating the workspace without the deployment_name? I suspect it could be the cause.

[BradSheridan]

Hi @Prabakar Ammeappin​ I relaunched the CFT and this time did not fill in the "(Optional) Recommended to provide a unique deployment name for your workspace" parameter but still no luck....

Received response status [FAILED] from custom resource. Message returned: Conflict InputData: {'workspace_name': 'databricks-nsc-poc94-workspace', 'aws_region': 'us-east-1', 'credentials_id': '6e78ed35-26c6-423b-b16f-711765fd2643', 'storage_configuration_id': '14797b70-932b-4f4d-b8c5-64f12711d877', 'network_id': 'fa8856e7-442a-44b1-9b42-16c8cc0c0545'} (RequestId: d819f683-574e-4342-b59d-d3072dcea787)

[BradSheridan]

Regarding how we overcame the copyObject error, my engineer responded saying that we were using a different (Databricks provided??) Cloudformation template until our Databricks Sales Engineer told us to use the AWS Quickstart template

[Prabakar]

Could be a conflict with the previous failed process. Check if there are old setups and delete everything. Allow 30 mins so all resources are cleared properly. Then try recreating the workspace.