Workspace deployment on AWS - CloudFormation Issue

[mihai]

Hello,

I have been trying to deploy a workspace on AWS using the quickstart feature, and I have been running into a problem where the stack fails when trying to create a resource.

The following resource(s) failed to create: [CopyZips].

From the CloudWatch logs I have gathered that the issue lies with a Lambda function that seems to call a CopyObject operation.

copy_source: {'Bucket': 'databricks-prod-public-cfts', 'Key': 'functions/packages/default-cluster/lambda.zip'}

Which fails with an AccessDenied Error as I do not seem to have access to that bucket from my AWS account.

[ERROR] 2022-01-21T15:48:42.138Z b49f0923-0c8b-4893-bf77-ae559c7fde88 Exception: An error occurred (AccessDenied) when calling the CopyObject operation: Access Denied
Traceback (most recent call last):
 File "/var/task/index.py", line 42, in handler
  copy_objects(source_bucket, dest_bucket, prefix, objects)
 File "/var/task/index.py", line 17, in copy_objects
  s3.copy_object(CopySource=copy_source, Bucket=dest_bucket,
 File "/var/runtime/botocore/client.py", line 386, in _api_call
  return self._make_api_call(operation_name, kwargs)
 File "/var/runtime/botocore/client.py", line 705, in _make_api_call
  raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the CopyObject operation: Access Denied

 If I am not mistaken, the databricks-prod-public-cfts bucket is managed by Databricks, and AWS users should be able to access it for sourcing the resources needed towards deploying a Databricks workspace, yet I do not appear to have access after creating my Databricks account.

[Anonymous]

Hi there! My name is Piper, and I'm a moderator for the Databricks community. I apologize for my delayed response! Let's give it a bit longer to see what the community has to say before we come back to you.

@Hubert Dudek​ or @Werner Stinckens​ - Can either of y'all help?

[-werners- · Accepted solution]

can you check the cross-account IAM role? According to the quickstart this is pretty important

[mihai]

The cross-account IAM role was created properly with this policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "ec2:AllocateAddress",
                "ec2:AssociateDhcpOptions",
                "ec2:AssociateIamInstanceProfile",
                "ec2:AssociateRouteTable",
                "ec2:AttachInternetGateway",
                "ec2:AttachVolume",
                "ec2:AuthorizeSecurityGroupEgress",
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:CancelSpotInstanceRequests",
                "ec2:CreateDhcpOptions",
                "ec2:CreateInternetGateway",
                "ec2:CreateKeyPair",
                "ec2:CreateNatGateway",
                "ec2:CreatePlacementGroup",
                "ec2:CreateRoute",
                "ec2:CreateRouteTable",
                "ec2:CreateSecurityGroup",
                "ec2:CreateSubnet",
                "ec2:CreateTags",
                "ec2:CreateVolume",
                "ec2:CreateVpc",
                "ec2:CreateVpcEndpoint",
                "ec2:DeleteDhcpOptions",
                "ec2:DeleteInternetGateway",
                "ec2:DeleteKeyPair",
                "ec2:DeleteNatGateway",
                "ec2:DeletePlacementGroup",
                "ec2:DeleteRoute",
                "ec2:DeleteRouteTable",
                "ec2:DeleteSecurityGroup",
                "ec2:DeleteSubnet",
                "ec2:DeleteTags",
                "ec2:DeleteVolume",
                "ec2:DeleteVpc",
                "ec2:DeleteVpcEndpoints",
                "ec2:DescribeAvailabilityZones",
                "ec2:DescribeIamInstanceProfileAssociations",
                "ec2:DescribeInstanceStatus",
                "ec2:DescribeInstances",
                "ec2:DescribeInternetGateways",
                "ec2:DescribeNatGateways",
                "ec2:DescribePlacementGroups",
                "ec2:DescribePrefixLists",
                "ec2:DescribeReservedInstancesOfferings",
                "ec2:DescribeRouteTables",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSpotInstanceRequests",
                "ec2:DescribeSpotPriceHistory",
                "ec2:DescribeSubnets",
                "ec2:DescribeVolumes",
                "ec2:DescribeVpcs",
                "ec2:DetachInternetGateway",
                "ec2:DisassociateIamInstanceProfile",
                "ec2:DisassociateRouteTable",
                "ec2:ModifyVpcAttribute",
                "ec2:ReleaseAddress",
                "ec2:ReplaceIamInstanceProfileAssociation",
                "ec2:ReplaceRoute",
                "ec2:RequestSpotInstances",
                "ec2:RevokeSecurityGroupEgress",
                "ec2:RevokeSecurityGroupIngress",
                "ec2:RunInstances",
                "ec2:TerminateInstances"
            ],
            "Resource": [
                "*"
            ],
            "Effect": "Allow",
            "Sid": "Stmt1403287045000"
        },
        {
            "Condition": {
                "StringLike": {
                    "iam:AWSServiceName": "spot.amazonaws.com"
                }
            },
            "Action": [
                "iam:CreateServiceLinkedRole",
                "iam:PutRolePolicy"
            ],
            "Resource": [
                "arn:aws:iam::*:role/aws-service-role/spot.amazonaws.com/AWSServiceRoleForEC2Spot"
            ],
            "Effect": "Allow"
        }
    ]
}

But it does not seem to have any bearing for S3 access, and the Lambda function that fails does not seem to depend in any way on this particular Role. The CloudFormation templates associates a different role to the CopyZips Lambda that includes the following policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::databricks-prod-public-cfts/*"
            ],
            "Effect": "Allow"
        },
        {
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::databricks-workspace-stack-lambdazipsbucket/*"
            ],
            "Effect": "Allow"
        }
    ]
}

[Atanu]

Could you please add CopyObject to the policy and see if that ease out the issue

?

[Kaniz]

Hi @Mihai Morar​ , Can you recheck if you're trying to access the right bucket?

AccessDenied indicates that you don't have permission so you need to work out if it's an API permission problem, a bucket permission problem, a coding problem, or problems with the environment that your code is running in (i.e. its effective credentials).

Might be worth making the source object world-readable and the target bucket world-writable temporarily and re-test. Then work up the chain from there.

[destepa]

Hi @Mihai Morar​  @Kaniz Fatma​  did you manage to solve this issue?

I am getting exactly the same error (Access Denied error in the CopyZips Lambda) when when creating the stack through the Quickstart, although the databricks-prod-public-cfts should be managed by Databricks and the rest of resources and permissions are created automatically by the template.

[Kaniz]

Hi @Daniel Estepa​ , Can you please check your Lambda privileges?

You can go to IAM dashboard, check the role associated with your Lambda execution.

[Anonymous]

Hey there @Mihai Morar​  and @Daniel Estepa​ 

Hope you guys are doing well. Just wanted to check in if you guys were able to resolve your issue or do you need more help?

We'd love to hear from you.

Thanks!

[GarethGraphy]

Dropping by with my experience in case anyone lands here via Google.

Note that the databricks-prod-public-cfts bucket is located in us-west-2.

If your AWS organisation has an SCP which whitelists specific regions (such as this example) and us-west-2 is not included, then the CopyObject action from the databricks-prod-public-cfts bucket in the CopyZips stage will fail.

Instead of adding us-west-2 to our list of whitelisted regions, I added s3:CopyObject to the NotAction list of exempt actions in the example SCP policy above. Then the SCP permits copying objects from a bucket in any region.

SCP errors manifest much in the same way as IAM errors, making them difficult to debug!

In an ideal world maybe Databricks would provide a mirror of this bucket in each region they operate in.