topic Re: Update existing Metastores in AWS databricks in Administration & Architecture

Update existing Metastores in AWS databricks

santosh23 — Wed, 18 Dec 2024 18:40:35 GMT

Hello Team,

I am unable to update the Existing Metastore in my AWS databricks.

I have new aws account and I am trying to update my existing workspace, however I am unable to update the s3 bucket details and Network configuration ( greyed out ) in the existing workspace.

Along with it I am unable to update the Metastore which is linked to the workspace with the new aws account.

So usually, how does it work.? how can I update the Metastore and workspace with the new AWS account .?

Re: Update existing Metastores in AWS databricks

Walter_C — Wed, 18 Dec 2024 18:56:23 GMT

This are the actual supported changes that can be run on a running workspace:

https://docs.databricks.com/en/admin/workspace/update-workspace.html#update-a-running-workspace

The Storage and Unity Catalog settings seems to not to be allowed to modify, and for network config you should be using Customer Managed VPC

Re: Update existing Metastores in AWS databricks

santosh23 — Wed, 18 Dec 2024 19:42:58 GMT

Yes, I tried updating the workspace with new configuration, however the storage configuration and metastore is not editable field ( Greyed out ).

And Also I am unable to updated the existing metastore with my new aws config, it is all greyed out.

Thanks.

Re: Update existing Metastores in AWS databricks

Walter_C — Wed, 18 Dec 2024 19:52:10 GMT

Yes because the workspace is in running state and this are not editable settings when the workspace is in running state only in failed as per the doc

Re: Update existing Metastores in AWS databricks

santosh23 — Wed, 18 Dec 2024 19:56:46 GMT

Is there a way to stop the workspace and update the settings ?

If there is no way to stop the workspace, how should I update my new aws account details to the existing workspaces .? should I create a new metastore and new workspace and then migrate all the data from old workspace to new workspace .?

Thanks.

Re: Update existing Metastores in AWS databricks

Walter_C — Wed, 18 Dec 2024 20:05:34 GMT

Let me confirm this to you but it seems that indeed you will need to create a new workspace, also please consider that you can only have 1 metastore per region in the account.

Re: Update existing Metastores in AWS databricks

santosh23 — Wed, 18 Dec 2024 21:29:37 GMT

Sure, Please let me know once you confirm.

Re: Update existing Metastores in AWS databricks

Walter_C — Thu, 19 Dec 2024 14:40:08 GMT

Unfortunately there is no way to move the state of the workspace manually so based on this the solution will be to recreate the workspace and migrate the data

Re: Update existing Metastores in AWS databricks

santosh23 — Thu, 19 Dec 2024 14:43:39 GMT

Sure, Thanks for the update.

is there any documentation where I can follow on how to migrate the data from one workspace to another workspace .?

Thanks.

Re: Update existing Metastores in AWS databricks

Walter_C — Thu, 19 Dec 2024 14:52:31 GMT

You could use the following migration tool: https://github.com/databrickslabs/migrate

Re: Update existing Metastores in AWS databricks

santosh23 — Thu, 19 Dec 2024 17:54:23 GMT

Thanks Walter for the migration tool. Does this migration tool can migrate all the Catalogs and SQL warehouse data from existing workspace to new workspace .?