cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Workspace deployment on AWS - CloudFormation Issue

mihai
New Contributor III

Hello,

I have been trying to deploy a workspace on AWS using the quickstart feature, and I have been running into a problem where the stack fails when trying to create a resource.

The following resource(s) failed to create: [CopyZips].

From the CloudWatch logs I have gathered that the issue lies with a Lambda function that seems to call a CopyObject operation.

copy_source: {'Bucket': 'databricks-prod-public-cfts', 'Key': 'functions/packages/default-cluster/lambda.zip'}

Which fails with an AccessDenied Error as I do not seem to have access to that bucket from my AWS account.

[ERROR] 2022-01-21T15:48:42.138Z b49f0923-0c8b-4893-bf77-ae559c7fde88 Exception: An error occurred (AccessDenied) when calling the CopyObject operation: Access Denied
Traceback (most recent call last):
 File "/var/task/index.py", line 42, in handler
  copy_objects(source_bucket, dest_bucket, prefix, objects)
 File "/var/task/index.py", line 17, in copy_objects
  s3.copy_object(CopySource=copy_source, Bucket=dest_bucket,
 File "/var/runtime/botocore/client.py", line 386, in _api_call
  return self._make_api_call(operation_name, kwargs)
 File "/var/runtime/botocore/client.py", line 705, in _make_api_call
  raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the CopyObject operation: Access Denied

 If I am not mistaken, the databricks-prod-public-cfts bucket is managed by Databricks, and AWS users should be able to access it for sourcing the resources needed towards deploying a Databricks workspace, yet I do not appear to have access after creating my Databricks account.

1 ACCEPTED SOLUTION

Accepted Solutions

-werners-
Esteemed Contributor III

can you check the cross-account IAM role? According to the quickstart this is pretty important

View solution in original post

7 REPLIES 7

Anonymous
Not applicable

Hi there! My name is Piper, and I'm a moderator for the Databricks community. I apologize for my delayed response! Let's give it a bit longer to see what the community has to say before we come back to you.

@Hubert Dudekโ€‹ or @Werner Stinckensโ€‹ - Can either of y'all help?

-werners-
Esteemed Contributor III

can you check the cross-account IAM role? According to the quickstart this is pretty important

mihai
New Contributor III

The cross-account IAM role was created properly with this policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "ec2:AllocateAddress",
                "ec2:AssociateDhcpOptions",
                "ec2:AssociateIamInstanceProfile",
                "ec2:AssociateRouteTable",
                "ec2:AttachInternetGateway",
                "ec2:AttachVolume",
                "ec2:AuthorizeSecurityGroupEgress",
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:CancelSpotInstanceRequests",
                "ec2:CreateDhcpOptions",
                "ec2:CreateInternetGateway",
                "ec2:CreateKeyPair",
                "ec2:CreateNatGateway",
                "ec2:CreatePlacementGroup",
                "ec2:CreateRoute",
                "ec2:CreateRouteTable",
                "ec2:CreateSecurityGroup",
                "ec2:CreateSubnet",
                "ec2:CreateTags",
                "ec2:CreateVolume",
                "ec2:CreateVpc",
                "ec2:CreateVpcEndpoint",
                "ec2:DeleteDhcpOptions",
                "ec2:DeleteInternetGateway",
                "ec2:DeleteKeyPair",
                "ec2:DeleteNatGateway",
                "ec2:DeletePlacementGroup",
                "ec2:DeleteRoute",
                "ec2:DeleteRouteTable",
                "ec2:DeleteSecurityGroup",
                "ec2:DeleteSubnet",
                "ec2:DeleteTags",
                "ec2:DeleteVolume",
                "ec2:DeleteVpc",
                "ec2:DeleteVpcEndpoints",
                "ec2:DescribeAvailabilityZones",
                "ec2:DescribeIamInstanceProfileAssociations",
                "ec2:DescribeInstanceStatus",
                "ec2:DescribeInstances",
                "ec2:DescribeInternetGateways",
                "ec2:DescribeNatGateways",
                "ec2:DescribePlacementGroups",
                "ec2:DescribePrefixLists",
                "ec2:DescribeReservedInstancesOfferings",
                "ec2:DescribeRouteTables",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSpotInstanceRequests",
                "ec2:DescribeSpotPriceHistory",
                "ec2:DescribeSubnets",
                "ec2:DescribeVolumes",
                "ec2:DescribeVpcs",
                "ec2:DetachInternetGateway",
                "ec2:DisassociateIamInstanceProfile",
                "ec2:DisassociateRouteTable",
                "ec2:ModifyVpcAttribute",
                "ec2:ReleaseAddress",
                "ec2:ReplaceIamInstanceProfileAssociation",
                "ec2:ReplaceRoute",
                "ec2:RequestSpotInstances",
                "ec2:RevokeSecurityGroupEgress",
                "ec2:RevokeSecurityGroupIngress",
                "ec2:RunInstances",
                "ec2:TerminateInstances"
            ],
            "Resource": [
                "*"
            ],
            "Effect": "Allow",
            "Sid": "Stmt1403287045000"
        },
        {
            "Condition": {
                "StringLike": {
                    "iam:AWSServiceName": "spot.amazonaws.com"
                }
            },
            "Action": [
                "iam:CreateServiceLinkedRole",
                "iam:PutRolePolicy"
            ],
            "Resource": [
                "arn:aws:iam::*:role/aws-service-role/spot.amazonaws.com/AWSServiceRoleForEC2Spot"
            ],
            "Effect": "Allow"
        }
    ]
}

But it does not seem to have any bearing for S3 access, and the Lambda function that fails does not seem to depend in any way on this particular Role. The CloudFormation templates associates a different role to the CopyZips Lambda that includes the following policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::databricks-prod-public-cfts/*"
            ],
            "Effect": "Allow"
        },
        {
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::databricks-workspace-stack-lambdazipsbucket/*"
            ],
            "Effect": "Allow"
        }
    ]
}

Atanu
Databricks Employee
Databricks Employee

Could you please add CopyObject to the policy and see if that ease out the issue

?

destepa
New Contributor III

Hi @Mihai Morarโ€‹  @Kaniz Fatmaโ€‹  did you manage to solve this issue?

I am getting exactly the same error (Access Denied error in the CopyZips Lambda) when when creating the stack through the Quickstart, although the databricks-prod-public-cfts should be managed by Databricks and the rest of resources and permissions are created automatically by the template.

Anonymous
Not applicable

Hey there @Mihai Morarโ€‹  and @Daniel Estepaโ€‹ 

Hope you guys are doing well. Just wanted to check in if you guys were able to resolve your issue or do you need more help?

We'd love to hear from you.

Thanks!

GarethGraphy
New Contributor III

Dropping by with my experience in case anyone lands here via Google.

Note that the databricks-prod-public-cfts bucket is located in us-west-2.

If your AWS organisation has an SCP which whitelists specific regions (such as this example) and us-west-2 is not included, then the CopyObject action from the databricks-prod-public-cfts bucket in the CopyZips stage will fail.

Instead of adding us-west-2 to our list of whitelisted regions, I added s3:CopyObject to the NotAction list of exempt actions in the example SCP policy above. Then the SCP permits copying objects from a bucket in any region.

SCP errors manifest much in the same way as IAM errors, making them difficult to debug!

In an ideal world maybe Databricks would provide a mirror of this bucket in each region they operate in.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group