topic Re: How to edit the bucket path for the Unity Catalog metastore when it is set up later in Data Governance

How to edit the bucket path for the Unity Catalog metastore when it is set up later

tyorisoo — Thu, 31 Oct 2024 04:02:32 GMT

How to edit the bucket path for the Unity Catalog metastore when it is set up later
Registered on 2024/10 from AWS marketplace.
We have created a customer management VPC and manually created the workspace.
I create S3 for metastore storage from workspace external location creation and set S3 path from account console.

Is there any way to change the bucket path and associated credentials?

Here is what we have tried
I could not edit the bucket from an external location.
I was able to modify another IAM role credentials, though,
The metastore screen of the account console did not reflect the changed IAM role.

Re: How to edit the bucket path for the Unity Catalog metastore when it is set up later

NandiniN — Thu, 31 Oct 2024 14:19:35 GMT

Hi @tyorisoo ,

Editing the Bucket Path for Unity Catalog Metastore

To edit the bucket path for the Unity Catalog metastore, follow these steps:

Prerequisites:
- Ensure you have the necessary permissions to manage the metastore and the associated S3 bucket.
Update the S3 Path:
- You cannot directly edit the bucket path from an external location. Instead, you need to update the metastore configuration.
Modify IAM Role Credentials:
- If you need to change the IAM role credentials, you can do so, but ensure that the new IAM role has the necessary permissions to access the new S3 bucket path.
Update Metastore Configuration:
- Use the Account Console or the Databricks API to update the metastore configuration with the new S3 bucket path and IAM role credentials.

Doc - https://docs.databricks.com/en/data-governance/unity-catalog/create-metastore.html#step-1-optional-create-an-s3-bucket-for-metastore-level-managed-storage-in-aws

Thanks!

Re: How to edit the bucket path for the Unity Catalog metastore when it is set up later

tyorisoo — Fri, 01 Nov 2024 00:39:29 GMT

Thank you.
What I was trying to explain is that, like this document(https://docs.databricks.com/ja/data-governance/unity-catalog/create-metastore.html)
External location (created manually), register your credentials and
after creating the S3 bucket from the account console,
I wanted to know how to edit it after creating the S3 bucket from the account console.

Can I edit the following after setting up as attached?
・change the bucket path
・Can the edited contents be reflected in the same IAM role?
・Change to a newly created IAM role

Here is what we have tried
・I could not edit the bucket from an external location.
・I was able to modify another IAM role credentials, though,
・The metastore screen of the account console did not reflect the changed IAM role.

Re: How to edit the bucket path for the Unity Catalog metastore when it is set up later

tyorisoo — Wed, 06 Nov 2024 00:55:06 GMT

@NandiniN
Sorry, I hope you are seeing this reply.

Re: How to edit the bucket path for the Unity Catalog metastore when it is set up later

NandiniN — Wed, 06 Nov 2024 04:40:25 GMT

Sorry @tyorisoo , I was not able to change the bucket. I will ask on of my team mates to review this and update here. Thanks!

Re: How to edit the bucket path for the Unity Catalog metastore when it is set up later

NandiniN — Wed, 06 Nov 2024 04:48:31 GMT

Is it however possible for you to create a new external location with the desired bucket path.

And for the metadata update - You should be able to modify the IAM role credentials https://docs.databricks.com/en/connect/unity-catalog/storage-credentials.html#step-3-update-the-iam-role-trust-relationship-policy as long as you have the sufficient permissions. Ideally changes should be reflected immediately, but it also could be a cache issue hence try to refresh/re-save the changes and review the logs.

Re: How to edit the bucket path for the Unity Catalog metastore when it is set up later

NandiniN — Wed, 06 Nov 2024 05:07:12 GMT

So @tyorisoo , I was able to check with one of my team mates. Only if you are an owner of the location, it can be edited, otherwise you ll have a to create a new one as I suggested earlier - create a new external location with the desired bucket path.

Re: How to edit the bucket path for the Unity Catalog metastore when it is set up later

tyorisoo — Thu, 07 Nov 2024 00:09:25 GMT

@NandiniN
Thanks for the reply.

I was able to check with one of my team mates. Only if you are an owner of the location, it can be edited, otherwise you ll have a to create a new one as I suggested earlier

I also created the external location.
Am I correct in that I can edit the path to an already created external location for the unitycatalog metastore?

What I want to know is on the account console screen of the captcha.
Metastore Settings.
How do I edit an already created external location for the unitycatalog metastore to change its path and role?

Re: How to edit the bucket path for the Unity Catalog metastore when it is set up later

NandiniN — Thu, 07 Nov 2024 10:38:00 GMT

Can you please try this -

-- Update the external location path ALTER EXTERNAL LOCATION my_external_location SET URL 's3://new-bucket/new-path'; -- Update the storage credential role ALTER STORAGE CREDENTIAL my_storage_credential SET IAM_ROLE 'arn:aws:iam::123456789012:role/new-role';

Re: How to edit the bucket path for the Unity Catalog metastore when it is set up later

NandiniN — Thu, 07 Nov 2024 10:40:05 GMT

Okay on carefully reviewing your image, doc and my internal repro

> You cannot modify this path once you set it.

https://docs.databricks.com/en/data-governance/unity-catalog/manage-metastore.html#step-3-add-the-storage-location-to-the-metastore