cancel
Showing results for 
Search instead for 
Did you mean: 
Data Governance
Join discussions on data governance practices, compliance, and security within the Databricks Community. Exchange strategies and insights to ensure data integrity and regulatory compliance.
cancel
Showing results for 
Search instead for 
Did you mean: 

Getting Access forbidden error to AWS when trying to follow the unity catalog quickstart notebook.

190809
Contributor

Hi

I have just created a metastore to allow unity catalog to be enabled for my workspace. However I am getting the following error when trying to create a table as per the unity catalog demo notbook:

Error in SQL statement: UnauthorizedAccessException: PERMISSION_DENIED: Access denied. Cause: 403 Forbidden error from cloud storage provider. exceptionTraceId=f5101e43-1f10-4e5f-8c15-0be034d7af8a

A couple of questions.

  1. If I delete the metastore will all of the data stored in my original hive-metastore be deleted ie. what existed before I created a metastore to enable unity catalog and assigned this to my existing workspace.
  2. Is there a way to re-configure the metastore to check that the credentials are correct?

Thanks in advance.

2 REPLIES 2

Sivaprasad1
Valued Contributor II

Hi,

Please hit metastore list and storage credentials API or CLI to get the detail of metastore and Storage credentials and you can verify the config.

https://api-docs.databricks.com/rest/latest/unity-catalog-api-specification-2-1.html

https://docs.databricks.com/dev-tools/cli/unity-catalog-cli.html#metastores

You can reconfigure your metastore or update the existing credentials:

To update the metastore with a new storage credential, you can use the following CLI commands.

AWS: https://docs.databricks.com/dev-tools/cli/unity-catalog-cli

Azure: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/unity-catalog-cli

databricks --profile test-shard unity-catalog storage-credentials create \

--json '{"name": "test-role-cred", "aws_iam_role": {"role_arn": "arn:aws:iam:::role/us-east-1-extdev-managed-catalog-test-role"}}'

databricks --profile test-shard unity-catalog metastores update \ --id bd5c8b05-xyzw-476a-8b4e-08320f9633d2 \

--json '{"storage_root_credential_id":"3783164d-9887-4b1c-afc6-a0b06cf3f44c"}'

Once the metastore is updated, check the below configs with the highlighted parameters:

Check the storage credentials config

databricks unity-catalog list-storage-credentials

The output should look like this:

{

"storage_credentials": [

 {

  "id": "9e3e6c22-e765-4da7-xxxx-***",

  "name": "4d173c3e-ee91-4c2f-a890-xxxxxxx-data-access-config-16607430xxxx",

  "aws_iam_role": {

  "role_arn": "arn:aws:I am:::role/my-unity-role",

  "unity_catalog_iam_arn": "arn:aws:iam:::role/unity-catalog-prod-UCMasterRole-14S5ZJVKOWXYZ",

  "external_id": "949b8e6e-f557-4aba-xxxx-***"

  },

  "owner": "kavya.parag+uc@databricks.com",

  "metastore_id": "4d173c3e-ee91-4c2f-xxxx-***",

  "created_at": 1660743064187,

  "created_by": "kp+uc@databricks.com",

  "updated_at": 1660743064187,

  "updated_by": "kp+uc@databricks.com",

  "used_for_managed_storage": true

 }

 ]

}

Check the metastore config:

databricks unity-catalog list-metastores

The output should look like this:

{

 "metastores": [

 {

  "name": "kp-test",

  "storage_root": "s3://bucket",

  "default_data_access_config_id": "9e3e6c22-e765-4da7-xxxx-***",

  "storage_root_credential_id": "9e3e6c22-e765-4da7-xxxx-***",

  "delta_sharing_scope": "INTERNAL",

"owner": "kp+uc@databricks.com",

  "privilege_model_version": "0.1",

  "metastore_id": "4d173c3e-ee91-4c2f-a890-***",

  "created_at": 1660743063329,

"created_by": "kp+uc@databricks.com",

  "updated_at": 1660743064838,

"updated_by": "kp+uc@databricks.com",

  "cloud": "aws",

  "region": "us-east-1",

  "global_metastore_id": "aws:us-east-1:4d173c3e-ee91-4c2f-xxxx-***"

}

190809
Contributor

Thanks for your answer to the second part of my question, super helpful! Can you confirm the first part of my question as well please?

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group