GarethGraphy
New Contributor III

Dropping by with my experience in case anyone lands here via Google.

Note that the databricks-prod-public-cfts bucket is located in us-west-2.

If your AWS organisation has an SCP which whitelists specific regions (such as this example) and us-west-2 is not included, then the CopyObject action from the databricks-prod-public-cfts bucket in the CopyZips stage will fail.

Instead of adding us-west-2 to our list of whitelisted regions, I added s3:CopyObject to the NotAction list of exempt actions in the example SCP policy above. Then the SCP permits copying objects from a bucket in any region.

SCP errors manifest much in the same way as IAM errors, making them difficult to debug!

In an ideal world maybe Databricks would provide a mirror of this bucket in each region they operate in.