Databricks Community

tyorisoo · 3 weeks ago

How to edit the bucket path for the Unity Catalog metastore when it is set up later
Registered on 2024/10 from AWS marketplace.
We have created a customer management VPC and manually created the workspace.
I create S3 for metastore storage from workspace external location creation and set S3 path from account console.

Is there any way to change the bucket path and associated credentials?

Here is what we have tried
I could not edit the bucket from an external location.
I was able to modify another IAM role credentials, though,
The metastore screen of the account console did not reflect the changed IAM role.

NandiniN · 3 weeks ago

Hi @tyorisoo ,

Editing the Bucket Path for Unity Catalog Metastore

To edit the bucket path for the Unity Catalog metastore, follow these steps:

Prerequisites:
- Ensure you have the necessary permissions to manage the metastore and the associated S3 bucket.
Update the S3 Path:
- You cannot directly edit the bucket path from an external location. Instead, you need to update the metastore configuration.
Modify IAM Role Credentials:
- If you need to change the IAM role credentials, you can do so, but ensure that the new IAM role has the necessary permissions to access the new S3 bucket path.
Update Metastore Configuration:
- Use the Account Console or the Databricks API to update the metastore configuration with the new S3 bucket path and IAM role credentials.

Doc - https://docs.databricks.com/en/data-governance/unity-catalog/create-metastore.html#step-1-optional-c...

Thanks!

tyorisoo · 3 weeks ago

Thank you.
What I was trying to explain is that, like this document(https://docs.databricks.com/ja/data-governance/unity-catalog/create-metastore.html)
External location (created manually), register your credentials and
after creating the S3 bucket from the account console,
I wanted to know how to edit it after creating the S3 bucket from the account console.

Can I edit the following after setting up as attached?
・change the bucket path
・Can the edited contents be reflected in the same IAM role?
・Change to a newly created IAM role

Here is what we have tried
・I could not edit the bucket from an external location.
・I was able to modify another IAM role credentials, though,
・The metastore screen of the account console did not reflect the changed IAM role.

tyorisoo · 2 weeks ago

@NandiniN
Sorry, I hope you are seeing this reply.

NandiniN · 2 weeks ago

Sorry @tyorisoo , I was not able to change the bucket. I will ask on of my team mates to review this and update here. Thanks!

NandiniN · 2 weeks ago

Is it however possible for you to create a new external location with the desired bucket path.

And for the metadata update - You should be able to modify the IAM role credentials https://docs.databricks.com/en/connect/unity-catalog/storage-credentials.html#step-3-update-the-iam-... as long as you have the sufficient permissions. Ideally changes should be reflected immediately, but it also could be a cache issue hence try to refresh/re-save the changes and review the logs.

NandiniN · 2 weeks ago

So @tyorisoo , I was able to check with one of my team mates. Only if you are an owner of the location, it can be edited, otherwise you ll have a to create a new one as I suggested earlier - create a new external location with the desired bucket path.

tyorisoo · 2 weeks ago

@NandiniN
Thanks for the reply.

I was able to check with one of my team mates. Only if you are an owner of the location, it can be edited, otherwise you ll have a to create a new one as I suggested earlier

I also created the external location.
Am I correct in that I can edit the path to an already created external location for the unitycatalog metastore?

What I want to know is on the account console screen of the captcha.
Metastore Settings.
How do I edit an already created external location for the unitycatalog metastore to change its path and role?

NandiniN · 2 weeks ago

Can you please try this -

-- Update the external location path
ALTER EXTERNAL LOCATION my_external_location
SET URL 's3://new-bucket/new-path';

-- Update the storage credential role
ALTER STORAGE CREDENTIAL my_storage_credential
SET IAM_ROLE 'arn:aws:iam::123456789012:role/new-role';