topic Re: Workspace deployment on AWS - CloudFormation Issue in Data Engineering

Workspace deployment on AWS - CloudFormation Issue

mihai — Fri, 21 Jan 2022 18:43:26 GMT

Hello,

I have been trying to deploy a workspace on AWS using the quickstart feature, and I have been running into a problem where the stack fails when trying to create a resource.

The following resource(s) failed to create: [CopyZips].

From the CloudWatch logs I have gathered that the issue lies with a Lambda function that seems to call a CopyObject operation.

copy_source: {'Bucket': 'databricks-prod-public-cfts', 'Key': 'functions/packages/default-cluster/lambda.zip'}

Which fails with an AccessDenied Error as I do not seem to have access to that bucket from my AWS account.

[ERROR] 2022-01-21T15:48:42.138Z b49f0923-0c8b-4893-bf77-ae559c7fde88 Exception: An error occurred (AccessDenied) when calling the CopyObject operation: Access Denied
Traceback (most recent call last):
 File "/var/task/index.py", line 42, in handler
  copy_objects(source_bucket, dest_bucket, prefix, objects)
 File "/var/task/index.py", line 17, in copy_objects
  s3.copy_object(CopySource=copy_source, Bucket=dest_bucket,
 File "/var/runtime/botocore/client.py", line 386, in _api_call
  return self._make_api_call(operation_name, kwargs)
 File "/var/runtime/botocore/client.py", line 705, in _make_api_call
  raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the CopyObject operation: Access Denied

If I am not mistaken, the databricks-prod-public-cfts bucket is managed by Databricks, and AWS users should be able to access it for sourcing the resources needed towards deploying a Databricks workspace, yet I do not appear to have access after creating my Databricks account.

Re: Workspace deployment on AWS - CloudFormation Issue

Anonymous — Tue, 25 Jan 2022 16:21:35 GMT

Hi there! My name is Piper, and I'm a moderator for the Databricks community. I apologize for my delayed response! Let's give it a bit longer to see what the community has to say before we come back to you.

@Hubert Dudek or @Werner Stinckens - Can either of y'all help?

Re: Workspace deployment on AWS - CloudFormation Issue

-werners- — Tue, 25 Jan 2022 21:13:34 GMT

can you check the cross-account IAM role? According to the quickstart this is pretty important

Re: Workspace deployment on AWS - CloudFormation Issue

mihai — Wed, 26 Jan 2022 12:09:08 GMT

The cross-account IAM role was created properly with this policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "ec2:AllocateAddress",
                "ec2:AssociateDhcpOptions",
                "ec2:AssociateIamInstanceProfile",
                "ec2:AssociateRouteTable",
                "ec2:AttachInternetGateway",
                "ec2:AttachVolume",
                "ec2:AuthorizeSecurityGroupEgress",
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:CancelSpotInstanceRequests",
                "ec2:CreateDhcpOptions",
                "ec2:CreateInternetGateway",
                "ec2:CreateKeyPair",
                "ec2:CreateNatGateway",
                "ec2:CreatePlacementGroup",
                "ec2:CreateRoute",
                "ec2:CreateRouteTable",
                "ec2:CreateSecurityGroup",
                "ec2:CreateSubnet",
                "ec2:CreateTags",
                "ec2:CreateVolume",
                "ec2:CreateVpc",
                "ec2:CreateVpcEndpoint",
                "ec2:DeleteDhcpOptions",
                "ec2:DeleteInternetGateway",
                "ec2:DeleteKeyPair",
                "ec2:DeleteNatGateway",
                "ec2:DeletePlacementGroup",
                "ec2:DeleteRoute",
                "ec2:DeleteRouteTable",
                "ec2:DeleteSecurityGroup",
                "ec2:DeleteSubnet",
                "ec2:DeleteTags",
                "ec2:DeleteVolume",
                "ec2:DeleteVpc",
                "ec2:DeleteVpcEndpoints",
                "ec2:DescribeAvailabilityZones",
                "ec2:DescribeIamInstanceProfileAssociations",
                "ec2:DescribeInstanceStatus",
                "ec2:DescribeInstances",
                "ec2:DescribeInternetGateways",
                "ec2:DescribeNatGateways",
                "ec2:DescribePlacementGroups",
                "ec2:DescribePrefixLists",
                "ec2:DescribeReservedInstancesOfferings",
                "ec2:DescribeRouteTables",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSpotInstanceRequests",
                "ec2:DescribeSpotPriceHistory",
                "ec2:DescribeSubnets",
                "ec2:DescribeVolumes",
                "ec2:DescribeVpcs",
                "ec2:DetachInternetGateway",
                "ec2:DisassociateIamInstanceProfile",
                "ec2:DisassociateRouteTable",
                "ec2:ModifyVpcAttribute",
                "ec2:ReleaseAddress",
                "ec2:ReplaceIamInstanceProfileAssociation",
                "ec2:ReplaceRoute",
                "ec2:RequestSpotInstances",
                "ec2:RevokeSecurityGroupEgress",
                "ec2:RevokeSecurityGroupIngress",
                "ec2:RunInstances",
                "ec2:TerminateInstances"
            ],
            "Resource": [
                "*"
            ],
            "Effect": "Allow",
            "Sid": "Stmt1403287045000"
        },
        {
            "Condition": {
                "StringLike": {
                    "iam:AWSServiceName": "spot.amazonaws.com"
                }
            },
            "Action": [
                "iam:CreateServiceLinkedRole",
                "iam:PutRolePolicy"
            ],
            "Resource": [
                "arn:aws:iam::*:role/aws-service-role/spot.amazonaws.com/AWSServiceRoleForEC2Spot"
            ],
            "Effect": "Allow"
        }
    ]
}

But it does not seem to have any bearing for S3 access, and the Lambda function that fails does not seem to depend in any way on this particular Role. The CloudFormation templates associates a different role to the CopyZips Lambda that includes the following policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::databricks-prod-public-cfts/*"
            ],
            "Effect": "Allow"
        },
        {
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::databricks-workspace-stack-lambdazipsbucket/*"
            ],
            "Effect": "Allow"
        }
    ]
}

Re: Workspace deployment on AWS - CloudFormation Issue

Atanu — Sat, 12 Feb 2022 16:28:54 GMT

Could you please add CopyObject to the policy and see if that ease out the issue

Re: Workspace deployment on AWS - CloudFormation Issue

destepa — Wed, 16 Mar 2022 08:15:18 GMT

Hi @Mihai Morar @Kaniz Fatma did you manage to solve this issue?

I am getting exactly the same error (Access Denied error in the CopyZips Lambda) when when creating the stack through the Quickstart, although the databricks-prod-public-cfts should be managed by Databricks and the rest of resources and permissions are created automatically by the template.

Re: Workspace deployment on AWS - CloudFormation Issue

Anonymous — Wed, 27 Apr 2022 16:39:59 GMT

Hey there @Mihai Morar and @Daniel Estepa

Hope you guys are doing well. Just wanted to check in if you guys were able to resolve your issue or do you need more help?

We'd love to hear from you.

Thanks!

Re: Workspace deployment on AWS - CloudFormation Issue

GarethGraphy — Wed, 08 Jun 2022 13:16:37 GMT

Dropping by with my experience in case anyone lands here via Google.

Note that the databricks-prod-public-cfts bucket is located in us-west-2.

If your AWS organisation has an SCP which whitelists specific regions (such as this example) and us-west-2 is not included, then the CopyObject action from the databricks-prod-public-cfts bucket in the CopyZips stage will fail.

Instead of adding us-west-2 to our list of whitelisted regions, I added s3:CopyObject to the NotAction list of exempt actions in the example SCP policy above. Then the SCP permits copying objects from a bucket in any region.

SCP errors manifest much in the same way as IAM errors, making them difficult to debug!

In an ideal world maybe Databricks would provide a mirror of this bucket in each region they operate in.