cancel
Showing results for 
Search instead for 
Did you mean: 
Data Governance
Join discussions on data governance practices, compliance, and security within the Databricks Community. Exchange strategies and insights to ensure data integrity and regulatory compliance.
cancel
Showing results for 
Search instead for 
Did you mean: 

AWS Unity Catalog S3 Bucket Security

viSual
New Contributor II

Hi,

I've got Unity Catalog working but i cant create an external Table.

We have several Workspaces (with customer managed VPCs), all with Private Link connected to the Control Plane. Our Data S3 Buckets are secured via Bucket Policy (in addition to KMS) so only connections from the Control Plane and our own VPC Endpoints can do something as described here:

https://docs.databricks.com/administration-guide/cloud-configurations/aws/customer-managed-vpc.html#...

we need the Control Plane in there because of Commit Service https://docs.databricks.com/administration-guide/cloud-configurations/aws/s3-commit-service.html

its working without unity catalog (writing into the hive_metastore catalog), but as soon as we try to write through unity catalog we get a s3 permission denied. we can get it working by disabling the commit-service related restrictions but then our buckets are completely open to the control plane.

So my Question, why does the control plane need to write directly to our buckets with unity catalog and why isnt our own cluster writing the data with the associated iam_role?

or does unity catalog with commit-service need some more exception than:

"arn:aws:s3:::<bucket-name>/*_delta_log/*",

"arn:aws:s3:::<bucket-name>/*_spark_metadata/*",

"arn:aws:s3:::<bucket-name>/*offsets/*",

"arn:aws:s3:::<bucket-name>/*sources/*",

"arn:aws:s3:::<bucket-name>/*sinks/*",

"arn:aws:s3:::<bucket-name>/*_schemas/*"

thank you in advance 😉

1 REPLY 1

Debayan
Esteemed Contributor III
Esteemed Contributor III

Hi, Could you please confirm if you have followed this: https://docs.databricks.com/data-governance/unity-catalog/get-started.html , also, please confirm the exact error received?

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!