topic Re: Cloudformation error when launching Databricks in AWS in Data Engineering

Cloudformation error when launching Databricks in AWS

BradSheridan — Thu, 25 Aug 2022 18:20:12 GMT

I've seen many posts here in the Community as potential solutions to this error, but none seem to be a solution for us. We are trying to launch the 14 day free trial of Databricks from the AWS Marketplace and are getting the error below. Moreover, our partner manager at Databricks still hasn't replied to us, and neither have we received an answer when we emailed partners@databricks, so our hope is that someone here has the answer 🙂 Thanks!!

[ERROR] 2022-08-24T17:54:17.899Z ea746964-04dc-4796-9f5f-472f9b96b32d Exception: An error occurred (AccessDenied) when calling the CopyObject operation: Access Denied

Traceback (most recent call last):

File "/var/task/index.py", line 42, in handler

copy_objects(source_bucket, dest_bucket, prefix, objects)

File "/var/task/index.py", line 17, in copy_objects

s3.copy_object(CopySource=copy_source, Bucket=dest_bucket,

File "/var/runtime/botocore/client.py", line 391, in _api_call

return self._make_api_call(operation_name, kwargs)

File "/var/runtime/botocore/client.py", line 719, in _make_api_call

raise error_class(parsed_response, operation_name)

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the CopyObject operation: Access Denied

Re: Cloudformation error when launching Databricks in AWS

BradSheridan — Thu, 25 Aug 2022 18:35:57 GMT

looking now...however, what exactly am I looking for in the policy to identify as potentially incorrect?

Re: Cloudformation error when launching Databricks in AWS

Prabakar — Fri, 26 Aug 2022 07:47:49 GMT

CopyObject operation: Access Denied -- looks like the permission to storage bucket is denied. Is this happening for all users or only while using specific IAM roles?

Also if you are using a cross-account role please check if the role is present in the bucket policy.

Re: Cloudformation error when launching Databricks in AWS

BradSheridan — Fri, 26 Aug 2022 11:30:30 GMT

thanks for the response @Prabakar Ammeappin . We have manually created the S3 bucket and are waiting for someone at our client site that has IAM priviledges (we are contractors to them and don't have that permission) to create the cross-account role. My hope is to report back here in a few hours with the solution 🙂

Re: Cloudformation error when launching Databricks in AWS

BradSheridan — Fri, 26 Aug 2022 15:59:56 GMT

Hi again @Debayan Mukherjee. We're still struggling to get the cloudformation template deployed and I'm not getting much assistance from our partner manager. any chance you have availability for a brief call?

Re: Cloudformation error when launching Databricks in AWS

BradSheridan — Fri, 26 Aug 2022 16:18:22 GMT

Morning @Prabakar Ammeappin same question for you that I just posted to debayan....We're still struggling to get the cloudformation template deployed and I'm not getting much assistance from our partner manager. any chance you have availability for a brief call?

Re: Cloudformation error when launching Databricks in AWS

BradSheridan — Fri, 26 Aug 2022 18:16:41 GMT

@Debayan Mukherjee @Prabakar Ammeappin got passed the CopyObject error message and now the CloudFormation template errors out near the end on 'createWorkspace'. Any thoughts?

Re: Cloudformation error when launching Databricks in AWS

Prabakar — Sat, 27 Aug 2022 03:30:55 GMT

Hi @Brad Sheridan our time zone are different and i am unable to respond to you immediately. It's good to know you resolved the copyobject issue. Please provide the steps that you used to resolve the issue so it would help others.

For the error at create workspace, what is the error you are facing? Please add the screenshot with the error.

Re: Cloudformation error when launching Databricks in AWS

BradSheridan — Sat, 27 Aug 2022 13:22:44 GMT

Hi @Prabakar Ammeappin and thanks for getting back to me. The createWorkspace error happens near the end of the Cloudformation template and here is the error:

Received response status [FAILED] from custom resource. Message returned: Conflict InputData: {'workspace_name': 'nsc-poc-workspace', 'aws_region': 'us-east-1', 'credentials_id': '1f6914f7-9c47-4bf3-a1ed-c87c8451bea0', 'storage_configuration_id': '8ecf6b3e-3e6d-4629-af96-8cb70311bad1', 'network_id': '259c58e2-78f8-4285-b798-69126d87133c', 'deployment_name': 'nsc-poc'} (RequestId: d52f686d-d71a-4fd7-b15f-c15ec16da57e)

The following resource(s) failed to create: [createWorkspace].

Re: Cloudformation error when launching Databricks in AWS

BradSheridan — Sat, 27 Aug 2022 13:23:32 GMT

regarding the copyObject error that we overcame, I'll ask one of my engineers on Monday how he got passed the error and report back here.

Re: Cloudformation error when launching Databricks in AWS

Prabakar — Sat, 27 Aug 2022 15:17:22 GMT

Hi @Brad Sheridan Could you please try creating the workspace without the deployment_name? I suspect it could be the cause.

Re: Cloudformation error when launching Databricks in AWS

BradSheridan — Sat, 27 Aug 2022 16:19:03 GMT

Hi @Prabakar Ammeappin I relaunched the CFT and this time did not fill in the "(Optional) Recommended to provide a unique deployment name for your workspace" parameter but still no luck....

Received response status [FAILED] from custom resource. Message returned: Conflict InputData: {'workspace_name': 'databricks-nsc-poc94-workspace', 'aws_region': 'us-east-1', 'credentials_id': '6e78ed35-26c6-423b-b16f-711765fd2643', 'storage_configuration_id': '14797b70-932b-4f4d-b8c5-64f12711d877', 'network_id': 'fa8856e7-442a-44b1-9b42-16c8cc0c0545'} (RequestId: d819f683-574e-4342-b59d-d3072dcea787)

Re: Cloudformation error when launching Databricks in AWS

Prabakar — Sat, 27 Aug 2022 16:48:26 GMT

Could be a conflict with the previous failed process. Check if there are old setups and delete everything. Allow 30 mins so all resources are cleared properly. Then try recreating the workspace.

Re: Cloudformation error when launching Databricks in AWS

BradSheridan — Sun, 28 Aug 2022 13:39:19 GMT

Hi again @Prabakar Ammeappin - I took a brief look this morning at some of the things the Cloudformation template spins up (VPC, Roles, Bucket) and didn't see any leftovers from the previous attempt, so although I didn't check for everything that gets spun up, I feel confident there are no remaining things. What I did find interesting this morning is that it appears I don't even have a subscription to Databricks through the AWS Marketplace any longer??? (see screenshot)

Could this potentially be the issue?

Re: Cloudformation error when launching Databricks in AWS

BradSheridan — Sun, 28 Aug 2022 13:45:53 GMT

oh, and when I goto the AWS Marketplace to 'manage' my Databricks subscription, I see this "Subscription linking in-progress" message.

Re: Cloudformation error when launching Databricks in AWS

Prabakar — Sun, 28 Aug 2022 14:20:37 GMT

The subscription shows the account is canceled. So yes its the cause.

Re: Cloudformation error when launching Databricks in AWS

BradSheridan — Sun, 28 Aug 2022 14:34:14 GMT

ah ha...we're getting somewhere @Prabakar Ammeappin !! So when I click on the "Sign Up" link in the Databricks Admin console image above, I get this:

When I submit that form, I get:

Seeing as how I can login on the Databricks https://accounts.cloud.databricks.com/ site, I guess I need to 'contact our sales team'??

Re: Cloudformation error when launching Databricks in AWS

BradSheridan — Sun, 28 Aug 2022 15:03:00 GMT

Regarding how we overcame the copyObject error, my engineer responded saying that we were using a different (Databricks provided??) Cloudformation template until our Databricks Sales Engineer told us to use the AWS Quickstart template

Re: Cloudformation error when launching Databricks in AWS

Prabakar — Mon, 29 Aug 2022 03:06:54 GMT

Yes. It's better you check with the sales team to understand what is happening here.

Re: Cloudformation error when launching Databricks in AWS

BradSheridan — Thu, 01 Sep 2022 18:30:26 GMT

Here are some answers:

copyObject error - we were using a Databricks provided cloudformation template but this error goes away when we use the AWS provided template

createWorkspace error - we had subscribed>unsubscribed>resubscribed to Databricks via the AWS Marketplace. This cause our Databricks account to get 'locked' in a Cancelled status. We simply signed up again under a different email and the cloudformation template launched without an issue.

All good...hope this helps out!