topic external location on unity catalog permits access to s3 in Data Governance

external location on unity catalog permits access to s3

amitca71 — Thu, 29 Dec 2022 10:53:17 GMT

Hi,

when i create external location on unity catalog. even if i dont provide any user grants, i have an write access to the s3 bucket from attached workspace.

i created external location xxxx
i didnt allow any permissions on it
on a workspace that is connected to the metastore:

df.write.parquet('s3://xxxx-data/amit/suppose_to_fail')

ended successfully

when i removed the external location, it was failing with AWS error of no permissions, so i'm sure the permissions were delegated from UC.

How do i prevent from users to be able to write to s3 directly when using UC (btw, in the past, i could control the write options on AWS IAM level. though doesnt provide full solution, as i need to grant write selectively)

Thanks,

Amit

Re: external location on unity catalog permits access to s3

Sivaprasad1 — Thu, 29 Dec 2022 11:35:34 GMT

@Amit Cahanovich : Could you please share your config details,

Which DBR version ?

Cluster has any instance profile?

Re: external location on unity catalog permits access to s3

amitca71 — Thu, 29 Dec 2022 11:49:00 GMT

@Sivaprasad C S 11.2 (includes Apache Spark 3.3.0, Scala 2.12)

Instance profile =None

Re: external location on unity catalog permits access to s3

Sivaprasad1 — Thu, 29 Dec 2022 18:15:14 GMT

what is the cluster mode?

Could you please run below command and share results

uc permissions get --external-location <externallocationname> --profile <databricksprofile>

https://docs.databricks.com/dev-tools/cli/unity-catalog-cli.html#unity-catalog-cli

Re: external location on unity catalog permits access to s3

amitca71 — Thu, 29 Dec 2022 18:19:00 GMT

{

"num_workers": 0,

"cluster_name": "xxxxxx",

"spark_version": "11.2.x-scala2.12",

"spark_conf": {

"spark.master": "local[*, 4]",

"spark.databricks.cluster.profile": "singleNode",

"spark.databricks.dataLineage.enabled": "true"

"aws_attributes": {

"first_on_demand": 1,

"availability": "SPOT_WITH_FALLBACK",

"zone_id": "us-east-2a",

"spot_bid_price_percent": 100,

"ebs_volume_count": 0

"node_type_id": "i3.xlarge",

"driver_node_type_id": "i3.xlarge",

"ssh_public_keys": [],

"custom_tags": {

"ResourceClass": "SingleNode"

"spark_env_vars": {

"DB_CLUSTER_NAME": "\"***_xxxx\"",

"DD_SITE": "\"datadoghq.com\"",

"DB_CLUSTER_ID": "\"***_xxxx\"",

"DD_ENV": "staging",

"PYSPARK_PYTHON": "/databricks/python3/bin/python3",

"DD_API_KEY": "3aa81ed18bc46a1f9cc425ee6c5ada78"

"autotermination_minutes": 120,

"enable_elastic_disk": true,

"cluster_source": "UI",

"init_scripts": [

{

"dbfs": {

"destination": "dbfs:/FileStore/utils/datadog-install-driver-only.sh"

}

"single_user_name": "xxxx@***.***",

"data_security_mode": "SINGLE_USER",

"runtime_engine": "STANDARD",

"cluster_id": "0915-152649-ox2wxwwz"

}

Re: external location on unity catalog permits access to s3

amitca71 — Fri, 06 Jan 2023 14:28:15 GMT

@Sivaprasad C S

databricks unity-catalog external-locations get --name lakehouse-input --profile DEFAULT

{

"name": "xxxx",

"url": "s3://xxxx",

"credential_name": "databricks_unity_catalog",

"read_only": false,

"comment": "xxxxx",

"owner": "xxxx@***.xx",

"metastore_id": "xxxxxxxx",

"credential_id": "94ce13xxxxxxxxx2e3545e5",

"created_at": 1663136630885,

"created_by": "xxxx.xxxx@***.***",

"updated_at": 1663136630885,

"updated_by": "xxxx.xxxx@***.***"

}Is it because i' m the owner of the credentials?