Getting Access forbidden error to AWS when trying to follow the unity catalog quickstart notebook.

[190809]

Hi

I have just created a metastore to allow unity catalog to be enabled for my workspace. However I am getting the following error when trying to create a table as per the unity catalog demo notbook:

Error in SQL statement: UnauthorizedAccessException: PERMISSION_DENIED: Access denied. Cause: 403 Forbidden error from cloud storage provider. exceptionTraceId=f5101e43-1f10-4e5f-8c15-0be034d7af8a

A couple of questions.

1. If I delete the metastore will all of the data stored in my original hive-metastore be deleted ie. what existed before I created a metastore to enable unity catalog and assigned this to my existing workspace.
2. Is there a way to re-configure the metastore to check that the credentials are correct?

Thanks in advance.

[Sivaprasad1]

Hi,

Please hit metastore list and storage credentials API or CLI to get the detail of metastore and Storage credentials and you can verify the config.

https://api-docs.databricks.com/rest/latest/unity-catalog-api-specification-2-1.html

https://docs.databricks.com/dev-tools/cli/unity-catalog-cli.html#metastores

You can reconfigure your metastore or update the existing credentials:

To update the metastore with a new storage credential, you can use the following CLI commands.

AWS: https://docs.databricks.com/dev-tools/cli/unity-catalog-cli

Azure: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/unity-catalog-cli

databricks --profile test-shard unity-catalog storage-credentials create \

--json '{"name": "test-role-cred", "aws_iam_role": {"role_arn": "arn:aws:iam:::role/us-east-1-extdev-managed-catalog-test-role"}}'

databricks --profile test-shard unity-catalog metastores update \ --id bd5c8b05-xyzw-476a-8b4e-08320f9633d2 \

--json '{"storage_root_credential_id":"3783164d-9887-4b1c-afc6-a0b06cf3f44c"}'

Once the metastore is updated, check the below configs with the highlighted parameters:

Check the storage credentials config

databricks unity-catalog list-storage-credentials

The output should look like this:

{

"storage_credentials": [

 {

  "id": "9e3e6c22-e765-4da7-xxxx-***",

  "name": "4d173c3e-ee91-4c2f-a890-xxxxxxx-data-access-config-16607430xxxx",

  "aws_iam_role": {

  "role_arn": "arn:aws:I am:::role/my-unity-role",

  "unity_catalog_iam_arn": "arn:aws:iam:::role/unity-catalog-prod-UCMasterRole-14S5ZJVKOWXYZ",

  "external_id": "949b8e6e-f557-4aba-xxxx-***"

  },

  "owner": "kavya.parag+uc@databricks.com",

  "metastore_id": "4d173c3e-ee91-4c2f-xxxx-***",

  "created_at": 1660743064187,

  "created_by": "kp+uc@databricks.com",

  "updated_at": 1660743064187,

  "updated_by": "kp+uc@databricks.com",

  "used_for_managed_storage": true

 }

 ]

}

Check the metastore config:

databricks unity-catalog list-metastores

The output should look like this:

{

 "metastores": [

 {

  "name": "kp-test",

  "storage_root": "s3://bucket",

  "default_data_access_config_id": "9e3e6c22-e765-4da7-xxxx-***",

  "storage_root_credential_id": "9e3e6c22-e765-4da7-xxxx-***",

  "delta_sharing_scope": "INTERNAL",

"owner": "kp+uc@databricks.com",

  "privilege_model_version": "0.1",

  "metastore_id": "4d173c3e-ee91-4c2f-a890-***",

  "created_at": 1660743063329,

"created_by": "kp+uc@databricks.com",

  "updated_at": 1660743064838,

"updated_by": "kp+uc@databricks.com",

  "cloud": "aws",

  "region": "us-east-1",

  "global_metastore_id": "aws:us-east-1:4d173c3e-ee91-4c2f-xxxx-***"

}

[190809]

Thanks for your answer to the second part of my question, super helpful! Can you confirm the first part of my question as well please?